What it takes to ship agentic AI into production

All right, I think this webinar has now started. So guys, I will slowly make my intro. I see people are joining and they keep on joining throughout the session. So to all of you, welcome to another series that we are hosting at the MicroOne. Very pleased that in the conversation we're going to have today, I'm hosting Jason and Imran, and I'm going to do some quick introductions now, briefly, from what I have gathered from learning about Jason and Imran, but also give them an opportunity to kind of tell you how they got into what they currently do themselves. So I'm Mark. I'm the chief economist at MicroOne. That's what I've been dedicating myself for a few months now besides my, my primary life where I'm an academic working across Northeastern and Harvard. What we are trying to understand today is really a question on agents. This is something that has already emerged in the series of Micro One more than once. We're trying to really get a better understanding. There is still, I think, a bit of a hype around agent or agentic AI. And so the best way for us to really kind of dissipate the noise is to engage with thought leaders who are helping us to understand pretty much where we are with the state of work across what the organization in which they're represented are currently doing, but in general as well, where the industry is heading. So for this, I'm super happy to welcome Jason Mays to our series and Imran Nazim. Now, Imran and I, we do share MicroOne as a background. We've been part of this organization. I think, Imre, you joined probably a few months after I joined. I joined, it was February 2025, and you joined probably at the end of 2025, if I'm not mistaken.

That's right.

Yeah, that's right. Right? And then, so we're gonna talk more about your profile as well. But Jason, look, when I was kind of looking at your work, so you are at Google in the Web AI as a Web AI lead.

Yeah.

You've been working in the field for a few years now. So you've been kind of exposed to the industry across the different evolution that the industry had in the last few years. You've been involved in different kind of pipelines across your career. And somehow, right, I think what is interesting is that you created a WebAI course that, you know, thousands of people have been trained on. You do talk a lot about WebAI, which is something we're going to talk about as well today. And what I really think is so interesting is that you kind of— you've been championing this client-side alternative to a traditional server-side model and the idea that somehow you are kind of looking at the web, not necessarily just the agents, but the web as a form of evolution into the agents. So we're going to start from you, Jason, before then doing the introduction to Imra. What has been your journey This is what I could gather when I basically tried to look at your profile. But tell us your journey from the person who has built it up, right? And it's also nice to welcome you to our series.

Yeah, thank you for having me on today. So I've been at Google almost 15 years now, and I've held various roles over that time. Background in web engineering, all that good stuff, UX design, front end, back end, all across the board there. But in 2015, I was working on a project before machine learning was even cool, and I had to try and recognize some soda cans for a client. And, um, uh, I was like, well, there's this thing called TensorFlow internally that was not released yet. Um, I was like, that's pretty good at recognizing cats, so maybe I can repurpose that to solve these problems. And to cut a long story short, 3 months later, after nerding out some research engineers, I figured out how to repurpose what they built to recognize, um, these soda cans. And It was really hard to do. So I turned that into a package solution that was on the cloud and I was able to then upload a video 20 minutes later, get a JavaScript API back that you could then send an image to and classify if the thing was in the image or not. Okay. And that was then turned into Cloud AutoML Vision, which is Google's first ever AI offering at scale. So that sent me down the AI train and essentially in 2017, um, we had things like TensorFlow.js and deeplern.js come out and that was the birth of web AI, which is client-side AI. And that's where you can take these machine learning models and run them client-side in the browser. Now, back in 2017, that was like toy models, like just visualizing how a multi-layer perceptron or neural network could work and, uh, see how it is actually sending the data through the model. But then people realized that this is actually useful and running stuff on the client side gives you privacy, low latency, cost savings, and all that other kind of adjacent stuff that's very useful for businesses. So TensorFlow.js evolved into something bigger. We had other teams join into the mix like MediaPipe who specialized in body understanding and models like pose estimation and segmentation, all that kind of thing. And then eventually fast forward to today, we can now run generative AI models entirely locally in the browser, like our Gemma models, which led me to my work early last year in 2025 where I made the first agents in JavaScript running entirely locally powered by a Gemma 2 billion parameter model back in the day. Now obviously things have evolved since then. We've got better models coming out all the time. We've got Gemma 3 en now that's multimodal. Um, but the principles are the same. You know, we can write orchestration layers and all that good stuff in JavaScript, not Python, and have the total privacy aspects. And what I'm focused on now with the W3C and others is, um, exploring the potential of things like WebMCP, which is inspired by MCP but designed for the web first. So it's basically how you, how you can register your website's existing functionality as tools that agents that might live in the browser or elsewhere could then leverage and enable the agentic internet. So that's basically me in a nutshell.

That's great. Well, just, it's fascinating and then I'm happy that, um, you really sharing, uh, your experience with us, uh, today as, as you've been pioneering in this field, I know for, for a few years and, uh, I think you're 15 years, uh, at Google. It's, it's a remarkable amount of time. Especially considering how much the company has evolved and changed, right, in 15 years. And this is something that we're gonna directly benefit, right, from the conversation we're having. So equally super excited, Imran, to welcome you to this. Look, I, as a disclaimer, I know you're doing this out of Dubai. There are lots of alerts right now. I was sharing that I was able to leave Dubai 2 days ago. And so I remember the alerts on the phone then. And in my case, the first time it happened at midnight 30, when you kind freak a bit more out, right? Because you don't know what's going to happen. So I appreciate that you're still doing this regardless of probably the fact that your phone has been alerting you as well, right? For, for maybe shelter. But that said, look, what is fascinating about your profile, Ibra, is that I think you were on the media for being probably the youngest PhD in— was it in the UK at the age of 22? Right. In theoretical astrophysics. Then I know what you and I have shared is as our Harvard kind of a shared space where you've been on a, on a postdoctoral fellowship at Harvard. You have worked for IBM as well. You've been teaching. And I think what I find your profile so unique, embrace this kind of convergence across astrophysics, applied mathematics, healthcare, AI, enterprise generative deployment. You're working in the industry, you work in academia, and I think even more fascinating for me that you decided then to become a researcher at MicroOne, joining our organization. And I think for, for anyone who joins your organization, we kind of discover this is an incredible place to be, right? So this is, I guess, what really excited me, that having talent like yours on board But it's the same as I was asking Jason. Tell us your story from the perspective of the person who kind of built that up. So what brought you to where you are right now? What was driving your interest? Yeah. And tell us a bit what has defined, right, your professional interests.

Yeah, no, that's great. And thank you for the introduction there, Mark. So sort of classically, I've always been interested in numbers. I'm a numbers person. So I've always had a core of mathematics, whether that's applied math, pure math. So at a young age, I found that I would like to focus in this direction, which led me to do an undergrad quite early. So I started at the age of 14 and then I graduated at the age of 18. I went straight to a PhD in theoretical astrophysics. So my thesis was on black holes. So once again, a lot of the math-type stuff there. So Throughout my journey so far and hopefully to come, I've always maintained an applied mathematics core, and that's something that I think is super helpful going between disciplines, whether that's healthcare, whether that's climate and sustainability, whether that's various forms of AI. They all share very similar aspects with respect to underlying mathematics. I've done several years in enterprise AI as well in terms of agentic development and deployment. So that's another very interesting area that I'm sure we're gonna talk about a lot today. But I think sort of coming back to what drove me to the career path, it's I like solving problems. And I think once you break down complex problems into their fundamental building blocks, it comes back down to the fundamentals. And I think the universal language for that is maths. So I'm a bit of a nerd in that respect. But yeah, that's, that's where my passion lies. And hopefully that continues.

That's fascinating. Look, it's very happy that somehow we can talk with you and that MicroOne was able to be an option for you to pursue at this stage of your career. So again, very, very nice to meet you, although virtually. So gentlemen, I have my opening frame for you, something that I've been thinking about and interested on that. It's really a statement that I don't have an answer for. It's really more about throwing out there to you. And see what you think. So my starting statement for both of you is the most authentic AI never makes it past the demo stage. So this bit of a hyperbolic statement. And I think that there are many obstacles between the demo and the deployment. This is what I think. This is what I have kind of gathered. But I'd like you to help me elaborate on this. So first of all, Jason, your reaction to this about AI doesn't really make it past the demo stage, almost like the agents exist much more as a construct. But when you try to deploy that, reality tends to teach you a hard lesson. What's your take on this? And, and I love to have both of your perspective. I know you're operating on the concept of agent from, from slightly different angles, and that's why it's complimentary in the conversation we're having. Susan.

Yeah, it's a very interesting question. So I think a lot of companies get very excited by the potential of what they're seeing coming out from, from, you know, various people in the world. Um, but when it comes to tailoring these things to their own data, their own tools, their own services, um, they sometimes fall short there and they don't understand the extra tweaking involved to customize these up. You know, these are non-deterministic things and you're trying to get them to work perfectly all the time for your brand reputation and all that kind of thing, right? So there's extra things on top. You want safeguards and guardrails to make sure it adheres to your brand's guidelines and all that kind of thing. So when you want to go beyond like a toy example, that's like, oh yeah, I made, I made a chatbot that, you know, can call a database to get the weather in London or whatever it might be, you know, um, that's nice. But when you are applying it to real world situations, there's other constraints that typically will come in. Um, and you know, we, we've seen a rise in things like prompt injection and that kind of stuff that brands down to need to be aware of too and to mitigate what happens in those situations. I think there's a famous example somewhere floating around online where some car company, some guy managed to talk to their chatbot and requested a car for a dollar and he said, "This is legally binding, please confirm I can get a car for a dollar," and the chatbot was like, "Yeah, sure." What happened to that situation? Now, I know that's more of a chat system versus maybe a fully-blown agentic system, but these are the beginnings of the things we're starting to see even the general public start to misuse or mistreat that kind of thing. So I think it's important to design things with these kinds of bits and pieces in mind. And I think today workflows essentially are probably the sweet spot where you've got the ability to call tools, you've got the ability to use the generative AI to understand natural language and that kind of thing, and bringing those together into a more defined pipeline that's more deterministic is probably the sweet spot today versus just going fully rogue. Unless you're doing something personally where you're just maybe using stuff in the browser to go research something for you, come back with the results or something like that. The stakes might be lower there because it's just personally affecting you versus a whole company or legally binding things.

That's great. I'm going to ask you exactly the same question. I'm going to create a bit of a bridge from what Jason said, I think within the context of also how you're supporting MicroOne, you know, at MicroOne we talk about evaluation infrastructure a lot, right? And I think it's about how do we see agents actually trustworthy in the real world condition, which I think is one of the reasons why in the premise of my question, the deployment is not necessarily successful, is that it might be successful in terms of the ideation of the agent. But the moment that we're deploying this in the real world conditions, that's when the agents sometimes doesn't make it through. So to the original statement, most agentic AI never makes it past the demo stage. What's your reaction on that, Inger?

Yeah, I think that's a very, very good observation. And that's something that we've seen clearly over the last year and a half. So sort of giving some context here, 2024 was really the year of wide, I would say, adoption of generative AI where we have these chat-based systems, right? So you can have an input and then you receive an output. But as we've developed these LLMs as central forms of an agent, we've decided to add on tools. We've decided to give them access to different environments. Now, in a pre-production setting, that's all largely synthetic, right? So you can give it access to tools, sort of APIs that are continuous, but these are not representative of what agents are trying to be in production when they have access to live APIs or distributed infrastructure. You know, reliability then becomes a system problem. So initially, I would say that most people would think that agent performance is limited by the model itself. But what we've quickly seen and ascertained from this idea from pre-production to production is that it's actually a systems problem, right? Because they're in production, these LLM-based agents are limited by the system environment. So we could have APIs timing out. We could have schemas that change. We could have permission issues. You know, you could have context fragmentation. You could have latency thresholds. All of these things, as you take it from a pre-production setting to a production setting, you have to account for. But the problem is, is that a lot of development happens in a pre-production setting. That goes back to your point there, Mark, of, okay, so now given that we try and put it into production, well then how can we do this monitoring, right? And that therefore comes into the idea of contextual evaluations, which is how can we then look to monitor these domain-specific agents And one of these ways is contextual evaluations. Essentially, it gives you an idea of the failure modes that arise. And that way you have a monitoring capability to say, okay, these are my issues, here's how I solve it. So that gives you sort of the baseline issues there.

Yeah. What is fascinating, Imran, is, you know, as we're doing this series, we having different people coming on stage. I remember we had a conversation with two leaders in the robotics field. And if you think about it, robotics is suffering of this exactly the same issue. We tend to watch incredible videos where this robot is doing great stuff, but when you try to run the robot in real life, then a sidewalk suddenly happens, right? Or the Wi-Fi stops working. And so I think we're understanding much more that in the pre-deployment stage, things, they tend to be measured in a certain way. But when they get deployed, the complexity of our realities where I think the solidity of the model is proven not necessarily in terms of the ideation itself, but in the application of the model in the real world, which, which actually I think that's probably just why a web-based AI agent could be a preferred solution. Which brings me to a question I'm trying to build now in the kind of building a bit of a, of a wall with different bricks you got to share with me. So, Jason, I— how would you suggest that we're using web AI agents to do exactly this? So really help with that level of, of decision that we want the agents to have so the client feels safe rather than having a degree of delegation which might end up in being reckless autonomy, which is what we're trying to avoid, right? We try to avoid agents that are so decoupled from a sense of common judgment, right, that we will consider to be reckless in real world. So how does a web-based, web-based solution prevent that level of recklessness?

Sure. I mean, a lot of this comes down to, you know, your orchestration layer and the model and how it's trained and that kind of stuff too. But just to give some background to the WebAI side here, um, so obviously WebAI allows us to run these generative models client-side using your own GPU. And we're starting to see the rise of the AI PC from Intel and Apple's M1 architectures and all that kind of stuff. So, you know, we're getting to this point where a lot of consumers have hardware that's good enough to run these models locally, especially if they're smaller models in like the 1 to maybe 7 billion parameter size. And if they're quantized, you can run these models quite nicely, very fast, via something called WebGPU in the browser. So that allows us to leverage the graphics card to its full potential. And coming down the line, we'll have WebNN, which will allow us to use the NPU on all of these devices too. And essentially, you know, With the advances of things like WebMCP, which is kind of like MCP that you probably all have heard of already, WebMCP allows you as the website owner to expose your existing functionality. So maybe your amazon.com, you've got the search function for your, for all the products. You can expose that as a tool that agents that might be living in the browser itself natively, like Chrome and others now have agents as part of the web browser itself, or maybe it's in a Chrome extension or something like that, you know, an agent living in that world, or, or maybe it's not even in the browser and you're able just to leverage various website domains' knowledge to your advantage. So essentially every domain in the world becomes a tool or a bunch of tools that if you're signed in, and this is the beautiful thing in the web browser, everyone is signed into the services they love and use daily. And the browser is like the one place where you can make requests to all those places without having to be asking for sign-in every single time. So if you're in an app in the native, you've got to request permissions to maybe sign into another app to, to call an API. But in a browser, you're already signed into all of those domains. So essentially you can imagine a local model having a personalized understanding of what you prefer to use. So maybe I'm signed into adobe.com, I want to use Adobe's creative tools when I'm editing images and videos and things versus something else. Um, and this allows a level of personalization that we've not really seen before because you can have all that context stored locally and, and kept private as well. So maybe you've also got like, um, personal information about your habits and things locally in the context of the browser there, and you're not sending that to cloud, but it allows the agent to take actions on your behalf in a way that you would've preferred to do based on what you've done before. So I think there's a whole realm of interesting area that, that's worth exploring. And of course there's hybrid models where you might combine stuff with cloud as well. There's, there's many ways we can do this, but I'm very excited about this potential, this agentic internet forming. Whereby we can kind of, you know, get more time back in our lives essentially by using the things we've already got and making us more productive.

If I can actually do a follow-up with you on this, just for me to understand, right? So when you talk about WebMCP, if I understand it correctly, you're talking this as an alternative to the Model Contest Protocol, right? Which is kind of what we have been using before. So if somebody asks you, what's the practical difference? right?

Yeah.

Um, using WebMCP, right? And, and, and how does it matter for agente production? Uh, I'm also thinking about maybe people are watching us that they are actually asking themselves these questions. Uh, what would you say?

Yeah, so the way I'd like to describe it, and you know, I'm definitely not the guy who invented all this stuff, but, um, the way I see it is that if you're gonna implement MCP, the traditional one that came out from Anthropic, you've got to be somewhat AI savvy and, uh, you need to go host an MCP server somewhere and all that kind of stuff, uh, for the things you want to expose. Um, it— right now every company in the world has a website already. So if you could just annotate your existing web markup, either in HTML attributes or programmatically via JavaScript, to expose the functions you've already got as tools, then that becomes a lot easier to deploy, um, for people who might be less AI savvy and just want to be part of the game of this agentic world that we're moving towards. Um, now WebMCP, whilst it's called WebMCP, is more focused on the tool registration part of that specification. So it doesn't have an exact parallel with MTP original, And the WebMCP spec is still being defined by the W3C, so it's a work in progress and we welcome feedback from people who want to give feedback there, of course. Mm-hmm. So this is early days, but we started to see previews go out. I think Chrome has a preview in Chrome Canary. We can start to play with these, these things and see how you could evolve your website to leverage this, to make it easier to have natural interactions with your website to get stuff done faster, which could be great for your end users. And also in the future opens up your website for agentic use in a more reliable fashion versus some of the other competitors we've seen who are trying to use screen understanding to interact with your website, which is a whole can of worms in my personal opinion. I think it's a bad way to go about it. If you can expose things more reliably, like this is the search function, it expects this as an input, it gets this as an output, the agent can leverage that much faster and use less tokens. To get the work done versus trying to think through how to interpret this screenshot and the DOM and kind of guess basically. So I think it'd be much more reliable in the future if we use something like WebMCP. Yeah.

Great, great. Fantastic. I appreciate Jason so much. And I'm also speaking on behalf of people that didn't know necessarily the subtlety, but you're helping us to really understand what's the value, right? And the benefit of this. I have a question for you, Imbra, that is kind of bringing back to a bit of what we do at MicroOne. And I'll start a bit of a, on a personal anecdote. So my, my training is in economics. So there is a mathematical background there. And I started the Micro One as I was a trainer for human data. And one of the things that I realized is that no matter how, and so the concept of rigor in the formal training tends to be driven by the model that you're trying to apply. But when you're trying to do training on the data, you realize that it's human expertise that tends to catch what the model does not catch. So in your experience, where— what kind of failure modes do you see in the existing benchmarks that humans are basically correcting, right, that we're basically rectifying? So do you find there is a typology of failures that are basically recurring more and more? And the reason why I'm asking this is that we're arguing a lot about the human data training. It is a necessary companion to the development of these models, right? But what does it mean for somebody like you who spent your entire career in testing rigorous models. So help us to understand that sense of where the benchmarks are failing us.

Yeah, that's a great question. So I think to begin with, benchmarks are typically done in isolation, right? So I think that's one issue where a lot of these people, you know, they look at benchmarks and say, okay, this model is better than this model. Okay, that may be okay in isolation. However, if you're putting that as a central head into an agentic system, it's actually the environment that you really need to have considerations of, right? So one idea of this is around drift, for example. I think one of the biggest issues of production AI systems is drift in general. So as agents interact with real systems, you know, whether that be tools, APIs, documents, users, the environment itself is changing continuously. So you're almost trying to optimize at moving goalposts. Right? And then ultimately you're trying to distill this type of knowledge into an agent. Now, if you take that to an enterprise setting, less than 1% of the enterprise data has been sort of taken up by these frontier models, right? So they may be good at generic operations. However, if you try and tailor them to enterprise-specific workflows or processes, you're going to have a real issue there. So I think one of the things that I'm really quite bullish on is this idea of human data. One of the ways is looking at failure modes that we, that we observe with these agents. So one of them is around tool use functionality. So whether that's Excel manipulation or PowerPoint. So typical things that are used in business workflows, there is associated failure modes with that. But then there's also associated failure modes with the domain. And this domain could be finance, it could be healthcare, it could be anything. So one of the ways that we can look to overcome this is via human data. Because ultimately, we want to distill that human-level workflow automation and process into those models. So the basic process is we first want to find those failure modes and then develop human data to overcome those failure modes and then train the model additionally with that data to overcome those failure modes. So it almost becomes this optimization loop. Now, if you can get that continuously, you're going to have not only an ability to monitor your agent, but also ensure it's performing as expected. And I think that's the type of area that we are in particular looking forward to. But I think in terms of the fundamentals, it's a really great area to move forward to in general.

Yeah. If I can also do a follow-up again, I'm kind of speaking, I'm voicing out what maybe other people might be asking. So So stress testing has been around for a long time. We understand the concept of stress testing, but what does it mean to stress test when we are dealing with a deployed agent? Because I think I understand stress testing if you're looking at the, the pre-deployment where you're kind of looking at the condition of where, for example, if you're looking against safety, you can determine whether or not a specific environment is safe or not. But how do you stress test when you already have a deployed agent? Is there something that, that, you know, you can share with us about that?

Yeah, that's a great question, Mark. So I think one of the actually the biggest gaps in current AI deployments is actually stress testing. So mostly this is done where agents are under ideal conditions. Right. And it's usually done at one time. So for example, in a pre-deployment stage, you're going to stress test your agent and say, okay, it's good, it's fit for deployment. However, this is a— you require a continuous monitoring and evaluation framework, right? These are not static systems. And I think this is one of the issues that we've seen in terms of production systems failing, where they're failing under these edge conditions, right? Where we have things like high latency, tool use failures, or unexpected inputs. So I think a lot of these static stress testing and pre-deployment is in order for that to be effective in deployment, We're gonna have to first make that static into continuous, and we're gonna have to get this continuous monitoring. So I think it's, it's a twofold problem. At the moment, it's done incorrectly in a pre-production setting in a static way. I think we're moving to a stage where this is gonna be continuous and needs to be continuous for, for deployment level production.

Right. Almost like the, the institutionalization of the, the feedback loop, right? In a way that it creates an ongoing, an ongoing understanding, which is, is strategic in the sense there is a calibration, but it's not calibration which is binary. It is actually evolutionary in this sense, right? Which is, I think, also from a concept perspective, more of a nonlinear model rather than thinking that it's just like, you know, A to B. And that's fascinating to hear. Jason, I'm going to bring it over to you. By the way, gentlemen, it's been over half an hour that we've been talking. I cannot believe it. It looks like literally 2 minutes ago that I introduced you. So, and look, if questions come, we'll see them in the Q&A. It's just I'm greedy now. I still want to keep you for myself and maybe ask you maybe one more question. So Jason, one of the things that I guess has been the reason for people to run an AI agent at the device level was because we want to create a sense of privacy. But in the moment in which we're thinking about the process shifting to the web, some might think, are we nonetheless incurring into a trade-off there? Is it a legitimate concern or is it more about we are extrapolating out of what we think could be a trade-off, but it's not a trade-off? Or is there a trade-off that you really need to make between using the agent on the web and maybe a dilution of privacy? What would be your take on that?

So you mean when we're using completely on the client side versus combined with cloud services at the same time? Yes. Okay.

Yes.

Yeah. So I guess in order to use various tools that may exist, those could be in the cloud somewhere, but the model itself and its knowledge about the end user could be local. So in that sense, it might know your preferences and how to leverage those tools without revealing personal information that you might have not wanted to share. Maybe You've got your finances listed somewhere and your spending habits, and it knows that you're gonna want the cheapest socks on Amazon when you go buy from Amazon without actually revealing that to Amazon, that that's your reason or intent, or giving up your pay history or something like that. So I think there's a nice balance there that could be complimentary whereby you can have a better agent experience because you've got that personalization without revealing that personalization to every service you interact with, if that makes sense. I think that's where the benefit comes from. Um, and also, as you, you know, in my vision of this agentic internet, as you browse websites, it learns tools that you probably care about. So instead of having this central repository of like millions of tools where you get this kind of paradox of choice, and we've seen after a certain tool count, it, you know, agents struggle to know which one to pick, right? So Um, you then get a more personalized collection of tools that you're probably more likely to use because those are the things you care about. That maybe I'm a photographer, I care about, you know, tools related to photography, and I signed into all of those kinds of sites, versus I'm probably not going to be using the tool for, I don't know, gaming or something like this. It might— yeah, you know, if I'm not that kind of personality. So, um, I think this could maybe allow a better agent experience, agent UX, whatever you want to call it, in the future by having that. I think that answers the question. Yeah, I think that— yeah, I think I'll leave it there for now.

It is just— you know, there's another quick follow-up, right? Like, you know, because I think when I was kind of studying your profile, that you are a certified information privacy technologist. Is that correct?

Yes. That was quite some time ago. But yes, I am. I am indeed.

Right. And I'm thinking if I kind of process this, you see privacy not just as a compliance but as a design constraint for agents, right? So how do you reconcile this, right? In the sense, how do you think of privacy when you're looking at your work? I'm asking you just to also expand on the definition of privacy from people who are working in the field, from maybe what we hear privacy maybe of a large language model, right? Or machine learning. Systems. So how, again, how did you get into that specific certification, right? That's out of curiosity. But then how do you think of privacy more than just past compliance?

I think the reason for having that certification is due to my very long tenure at Google and holding different jobs. At some point I was required to like prove that I could make web systems that, you know, respected these principles, especially when working with customers over in Europe, and there's more stringent kind of regulations over there. So it was a tick in the box back then. But yeah, I think some of this stuff has trickled over into my web AI work, and why I've kind of bet my career on some of this stuff. We're starting to see the industry generally care more about these things. You talk to people about how they're using AI today, and, you know, even I think in Apple's campaign, like privacy is like one of the things on their billboards, and security and all these other things, right? So, so this is the very top of mind right now for, for humans, generally speaking. Um, and I think it's, you know, good to be in control of when you want to reveal stuff to different services because, you know, what if there's a malicious agent out there and it just has free rein on all my data? I don't think I would want that. I want to be in control what I reveal and when. And design the systems responsibly to be like, at least be opt-in if they want to reveal certain aspects. So, you know, I could imagine in the same way as when we go to a website that wants your location, it asks you permission first, like, this website would like to know your location. I could see a future internet whereby, you know, this website needs to understand your financial history to be able to answer the question that you've asked it, and then you, you, you opt in to revealing that data. And then the agent can leverage that to give you a better answer, a more informed response or something like this with the tools that are exposed by that website or whatever it might be. But I think giving the user the option to own that is important. Yeah. Great.

Guys, this has been great. Look, there are questions that are coming in and I wanna make sure that we also address this. I had a final question from my side to you, Imran, before we go maybe to the question that the, our attendees have been putting in the chat. You know, I think we do both trust in human in the loop as an important concept, right? Is there a point where you sometimes see human in the loop as a bottleneck rather than a safeguard? And again, it's a question that I, I wouldn't know how to answer myself, but I've been asking myself this question, right? So do you, do you think there is this moment where you say this process of continuous evaluation where we are relying on people is creating a bottleneck rather than creating the evolution moving forward?

Yeah, that is a really good question. So I would say it can be a bottleneck for certain areas. I think one of the main ways it could be a bottleneck is in very deterministic settings. So for example, we know in maths 2 2 4, right? We don't require a human in the loop to do that because if we do use a human in the loop for those deterministic things, we run into a scalability problem, right? Yeah. So I think if we can harness the verifiability aspect of deterministic workflows more towards that. I think that optimization loop can be more well constrained. I think human in the loop is still very essential for subjective-based tasks so that whether that could be, you know, law or potential certain fields in economics, I think at this stage I'd be more comfortable with that. But we certainly have to make sure human in the loop is used in the right domains. And that way it's not such a bottleneck. We can reduce it somewhat.

Yeah. So somehow it's also reframing it not as a universal value that is applicable to everything, but also make it strategic in the sense, right, on where we need it rather than where we don't. It's like understanding the severity of the human in the loop based on the task that I'm trying to train rather than just as a universal concept, which is fascinating to hear, Imran. So, gents, look, I think there is some question. Sydney always helps me to distill it, and so I don't have necessarily to read it through directly. There was a question from David on WebAI that I'm going to go and address to you, Jason. I'm going to kind of read the version that Sydney has helped me synthesize. So will WebAI be fast enough to analyze multimodal content in the browser in real time to enable parental control filtering? And could Chrome have a few milliseconds of predictive buffering to make that feasible? So this is from David who is asking this question.

Yeah. What do you think, Josephine? Great question. So in terms of real-time multimodal models running client-side in the browser, yes, we have that today as long as your graphics card is, you know, fast enough. So, but if you've got like, you know, an NVIDIA-class graphics graphics card or the Apple M architecture or even Intel's latest built-in GPUs, integrated GPUs, they are fast enough to run these models in real time. There's a demo somewhere bashing around online with our Gemma 3N model running in real time on a webcam feed explaining what it sees. So yes, real time is possible and it's mind-blowing. Just like 2 years ago we were like like taking, I know, 10+ seconds to run Stable Diffusion in the browser, and now it's like 300 milliseconds or something. So all of these things are becoming, uh, possible.

Yeah, right. And so just since we are on with you, there is another question from Mohamed that Sydney helped me to synthesize. He's asking, does exposing MCP, uh, tools directly to browser-based agents increase the browser attack surface by allowing malicious sites to provide tools that could exfiltrate user data or manipulate agent decisions, which I think is building on the question on safety of running an agent on the web. And I think it's a great question that is speaking out for many of us asking this question.

Yeah, safety is obviously top of mind. And right now we are just in the creating the spec phase. We've not published the specification for WebMCP yet. It's an evolving spec with the W3C, Microsoft, Google, and Amazon individuals are all involved in this, and many people who are passionate can contribute too. And you know, one of the things that's actually come up in our recent discussions is like, okay, how do we prevent those kinds of attacks from happening? I believe right now the experiments that people are doing are more self-invoked, so you are, um, able to interact with the tools on a specific domain to test that they work. And so you are basically already there with your known extension that you want to use and you invoke it to then leverage the domain's tools to get stuff done faster. Um, once you start having cross-domain tool calling whereby you might not be currently on the site, but you're, you're leveraging all the other stuff that you've seen from the past, then you're totally correct. We need to start thinking about permissions to leverage that cross-domain stuff. So maybe it's like, hey, in order to complete this task that you've asked of me, I'm going to need to leverage your signed-in Amazon account to order that product. Do you allow it? You know, and yeah, these things, how that agent UX is revealed to the user in the context of web is still up for discussion, but these are things people are thinking about right now. If you've got opinions on this, we welcome you to come to the W3C GitHub for WebMCP and please weigh in on these. I think there's a bug open for security concerns and things like that that people can weigh in on and give their feedback there in case we've overseen anything there. That'd be great to get your—

David was very happy with your answers. I wanted to say thank you. I play the moderator in this case, but I appreciate this. This answer to you. Imre, there's a question which was the first one. Nathan was the one that wrote it in the chat. And what I find fascinating was the fact that he was basically asking the question, how do we kind of know, right, that somehow an output that is coming from another large language model, which of course could become an agent, right, is kind of a new combinatorial way of looking at the question I'm asking versus something that is disrupting my expectations. So how do you find a balance between what we know and we want to assess versus what we don't know but is equally happening because of this actually mathematical combination that we have, right? How do you navigate that? I find it to be almost like a perfect storm, right? In many ways. Because this is creating, striking a balance that we don't necessarily know what we are striking with, right?

Exactly. Great question, Mark. So I would say when we are looking to evaluate model outputs, I think one thing we should be careful with is not to enforce a single correct answer if the problem itself allows multiple valid solutions. I would say a good evaluation framework you can separate sort of the correctness from agreements with the evaluator. So instead of asking, for example, is this the answer that I would give, you can sort of ask a few different questions such as, is the reasoning valid? Does the solution satisfy the constraints? Or is it internally consistent or supported by some evidence? So I think oftentimes we just want one single correct answer, but if the problem itself allows multiple solutions, I think then we have to look more towards reasoning validity constraints, and any evidence that does support that.

Yeah. And there's a question. Angela is also training at MicroOne. I think she asked a question that is interesting, right? So you were mentioning before about kind of the legacy of the firms, the way they work. And so if you had to suggest or advise how we are building this ongoing evaluation of the models, right, would you see this as in-house experts So internal capabilities, would you see this more about an outsourced function to a technology company is gonna look after this? So how do you envision this optimization loop in reality? If a company is asking, how do I do that? What would be your, your take on that?

That's a great question. I would say to, to get people internally to do it is quite difficult because they already have day jobs themselves. Right? And there could also be some internal bias associated with things. So in terms of getting that human data to overcome those failure modes, I think it's quite important to actually outsource for that human data itself. Oftentimes you may think, oh, it's just easier to do in-house. But the people that you have working, they're already doing a job. So unless if you're bringing more and more people on to do this particular task and then build everything in-house, it's quite a laborious thing to construct. So I would say that while yes, it may seem logical to get a couple people to develop the human data, to do this in a scalable way, you're going to have to outsource. I think that's just one thing that's going to happen. Yeah.

Uh, gentlemen, we are actually getting at time, right? And I, I want to keep it within our kind of 45 minutes time frame. If you allow me the very final question to one of you from my side. Um, I, I would really appreciate that. Just my question as kind of our, my closing question to you, right? Uh, you kind of had described to us also through your work as an agentic internet, right? Where the website are basically serving both humans and agents, right? So if you're looking at the next few months, uh, what has to be true for this to happen responsibly? Because I think we are not fully there yet, although I think I think you are working in that direction. So help us to see the road ahead if we want to get to the point where you have an agentic, basically, web.

Yeah. As already mentioned, the security aspects are very important here. You want to maintain control there as an end user, not to like have some website that's trying to invoke the agent on your behalf to go do things. Instead, I think it should be human-driven, at least initially, so that you demand something to be done, the agent, the orchestration there comes up with a plan of what tools it needs to call to get the task done, request permissions from the user, say yes, go ahead, do this, boom, and then off to the races you go. Now moving towards the future, I could see this evolving to be even more advanced in the fact that maybe someone can just say what they want, and the browser is really minimal. You're not even going to the domains, you just signed into them, and you have this whole agentic UX coming back, which is almost tailoring the content based on your preferences. Maybe I'm a more visual person, it gives me graphs and charts, and maybe I'm a more analytical person, gives me tables of all the juicy numbers. Um, so I think that's a whole area of research that's worth exploring too, and how we can brand those experiences so you don't lose the brand reputation with the brands that are providing the information back to you. So how that can be customized up, but in a way that's flexible enough to work for anyone. And that's a really hard problem to solve well, but I know that's an active area of research going on within the wider community and how that can be achieved. And I think that leads to a very interesting possibility of what could happen there. 60 6% of time is spent in a browser according to Intel, I think it was. So, um, if people are already there, how do we meet them in the middle to allow them to get more time back in their lives and get jobs done, but also allowing them to discover things on the old internet with the old UIs and things? You know, you want a nice balance there. And, you know, I don't think anything's going away overnight. You know, people like granddads like me are still going to want to see the UI of the old days to kind of click on things and do stuff every once in a while. But there'll be certain things we delegate and as our children grow up, they'll expect that delegation from day one. And like, why can't I just say something and have it done for me versus actually, you know, being more manual in the process. So I think that's how I see things evolving there. Yeah.

You know, one of the things that I learned from you today, Jason, is that, you know, if we were thinking that Web3 is the, this is a transition towards more blockchain, maybe Web3 is more about an agentic internet. Rather than, than what was the original idea.

I like your thoughts there. I, I, I'd say, um, web AI is the real web3.

Web AI, yes.

Um, it can provide tangible, tangible benefits today. Exactly.

Yeah, that's right. Imre, last question to you. Uh, moving, if you think about the next few months, right? We can even take the remainder to 2026. Uh, so what do you think a production-grade evaluation, uh, standard could look like for agentic AI. So how would you like that eventually to evolve in the next few months?

Yeah, I think the first thing that needs to happen there is continuous monitoring and then defining what to monitor, right? Because these agents have very different workflows that they are trying to optimize for, tasks to perform, business workflows to do. So understanding what you want to monitor and then how you want to monitor it, I think is the first step. And then that monitoring gives you an abundance of information about what's going well and what's going wrong. And then how that you, you end up fixing that is its own problem itself. And that says the human data element that I mentioned before, Mark, of we want to distill the correct workflows and processes and tasks from enterprises into these agents. And that's sort of the second step there. So what I would like to see over the next couple of months and sort of going on towards the end of 2026 is these developments that are starting to happen. So the first thing is around the continuous monitoring element. Element, and then comes in, well, once you've monitored something and you know what you want to monitor, how do you fix it when it goes wrong? Because eventually, and from what we've discussed, there's various forms of drift, et cetera. So I would like to see that bundled in a nice sort of productized way and seeing that widely adopted. I think that would be a great stance to have at this stage.

Great.

Somehow, you know, in your answer, Imran, you kind of addressed some of the very last question that Alvaes had mentioned in the chat. So I won't be asking the question, but I think you were able to kind of COVID a good part of that. Gentlemen, this has been— first of all, I cannot believe that 50 more minutes have gone by. Thank you so much for having made this such an interesting conversation. As I mentioned before, I do benefit the most from this conversation because I get to know you and I get to learn from you. Really appreciate the work you're doing in your respective fields in advancing you know, a responsible agentic web that is going to support more and more people who are already on the web, Jason and Imran, the ability for us to think of evaluation with rigor, but not in the traditional or more legacy sense of rigor, but more of a continuous sense of rigor that's evolving. And I appreciate the fact that we're colleagues at MicroOne. So it's nice that through this series I can get to know incredible people. And if you make it to Tokyo, I should be there as well for the next few days. So please ping me on Slack. Guys, thank you for this. We had an average of about 100 people always connected, plus people that will watch us. It's been an absolute pleasure to learn from you. I wish you, wherever you are, in San Francisco, in Dubai, a good continuation of day or a good evening. Thank you again for making this possible, and I hope to meet you in person. That's the only thing that sometimes the web conferences do not— does not really do. But it's just a matter of time when that will happen in real life. Thank you so much, guys. I appreciate meeting you and see you next time.

Cheers. Thank you. Thank you. Bye-bye.