How Companies are Actually Putting AI Agents into Production with OpenRouter's COO

questions there as well. And without further ado, I'm going to turn it over to you, Chris.

Cool. Uh, hey everyone, good to meet you. I'm Chris. Uh, uh, as David said, I am the co-founder and chief operating officer here at OpenRouter. Um, going to share, uh, some data that we have and sort of the story that we've seen play out over, uh, you know, really primarily the past 2 quarters in terms of, um, some of what we've seen around like the telltale signs that agents are making their way from sort of somewhat theoretical to very practical in a number of companies and just sort of share some data and our perspective. First, to talk a little bit about OpenRouter, what we do and why we have some of this information. We are the world's largest AI gateway. So we work with about 70 different cloud providers, model labs, sort of you name them, we work with them, right? Including the major closed source labs like Anthropic, OpenAI, Google, Mistral, Meta, xAI, all of them are close partners of ours, as well as huge numbers of open-source neoclouds, some more esoteric hosting providers, and ultimately give access to hundreds of models through tons of different clouds across lots of geographies with lots of features and normalize that all down to a single API, to a single billing relationship, to make it really easy to use the models that you need. We manage quota and capacity for you. We handle all sorts of availability and uptime things. We make sure that tool calling works exactly the same way everywhere. And this has become really, really popular with developers. And so many, well, a few million developers have tried the platform all over the world. And we are a very pure-play sort of like PLG tech company. We're now, you know, working our way up in sort of mid-market enterprise and enterprise. But really are very much like bottoms-up developer tooling is the way that we get adopted. And it's been just a super fun business to get to sit at the intersection of all of these different AI trends and see how LLMs in particular are being used all around the world and how companies are making use of them internally. So we have a real bird's-eye view of a lot of things. We process literally like a trillion plus tokens a day. And for anyone who looks at their own internal AI usage, a trillion tokens a day is like very legitimate volume. It's like very, very, you know, a real player on sort of a global scale. We're across all geographies, all countries, and we monitor all this stuff in real time, which is really fun. So we get to see like who's up and who's down. And, you know, you may see one purported latency number that this provider is the fastest or the slowest or the most expensive or the least expensive. And All of this is available in real time on OpenRouter. We monitor hundreds or thousands of API endpoints and manage all of this stuff all over the world and work with these companies all the time to say like, hey, there's a, you know, a quirk in your API or this thing's not behaving the way that we expect. And we have a ton of quantitative data across all this, but we also have a lot of qualitative data because we understand what the different use cases are. And so a key belief at OpenRouter is that, you know, as we think about different models and how they're being used and, you know, you have Sodnet and Flash and you have free models and open source models and closed source models, like the ultimate benchmark to understand what model should I be using for what is actually what they're actually being used for in the real world. So if you go to OpenRouter and you look at our rankings, we sample and anonymously categorize a ton of prompts securely on our infrastructure. And then aggregate statistics around which models are being used for which prompts, whether it's marketing automation or programming or scientific usage or financial advice. And we publish a lot of this data. We have a lot more of it internally that allows us to make good recommendations and help individuals, companies, enterprises pick models and providers that are relevant for them and make sure that they can implement the inference they need across their entire organization. So super fun place to work. We have tons of data internally and we get to like sort of see trends play out on a global scale. We published a State of AI report at the end of 2025 that you can find on our site and I think got some, some good press that explains how some of this, some of these trends came together and happy to share some of those with you. Like I said, we process trillions of tokens a week and we've been growing pretty quickly and continue to grow. Have really material spend into the hundreds of millions of annual AI spend across all of these clouds. So have some real scale that's been built over the last 2 years as the company has really performed well. So it's been really fun. OK, so what do we see? We get to look at all kinds of cool stuff. So here is a look at the last— I guess this is actually— is this 2 years? I guess I should have looked at this chart more closely. It looks like it goes back to 2024 of closed versus open source model adoption. By the way, we do have the chat open. I have it open on my screen. So if you have questions about this stuff or something I said doesn't quite click, by all means, put it in the chat and I'll do my best to respond. But one of the things that we've been looking at is, and I think a lot of people wonder about, is closed source versus open model adoption. And if you look, you can kind of see that the open source proprietary split has actually been pretty steady for really quite some time. The real story here is the rise of Chinese open weight models, which is the lighter blue at the bottom of the screen. You can see that that's actually taken dominant share from a lot of US, primarily at least like US open weight models. Mistral has another of these that are quite good as well. But the rise of the Chinese open weight models, we can see sort of coming on strong here. And by the way, these are used heavily, perhaps most heavily by the US. US firms disproportionately use in particular the Quen family of models quite heavily in the Gentec flows. And firms like Minimax have come out with sort of agent-focused models. And we'll talk about sort of the role that those have to play in just a second. So when we talk about agents, we are talking about tool calls. And for some people this will be like, yeah, obviously, Chris, I know exactly how this works. But people will be like, wait, I didn't quite realize exactly how tool calls work. But when you think about agents, like, a simple way of thinking about them is it is a, a large language model in a loop interacting with tools, interacting with the outside world. And an interesting property of an LLM is that inherently they can't do anything, right? All they get to do is they take text in and they output text. That's it. They don't inherently have the capability to, to use tools or interact with the outside world. So what actually happens when we use tool calls with LLMs? Well, the loop looks something like this. A software harness or even a person interacting with an LLM says like, hey, tell me if this is a valuable customer. That actually is my email if you want to email me. You have some tools available like Stripe or Hacks, which is an analytics tool. Obviously, this is being done in— well, not in English, but you can sort of conceptually actually do it in English because LLMs are programmable in English. And the large language model comes back and says, hey, great. Please, I think the Stripe tool makes sense. Please call that for me. 'Cause the LLM can only output text. It can't call a Stripe tool. It can just say, here's some, here's some text that's a function call, please do it for me. The software harness executes it and says, okay, great. Hey, LLM, here's the result of that tool call. Here's the, here's the Stripe data. And the LLM says, okay, great. Based on that, I can see the user has spent $500. Sounds valuable. And each of these times it provides what's called a finish reason. So you can see already this is a loop, right? A software harness is looping over multiple LLM calls. And an agent is doing this more and more times. And it is a looping call over LLMs with tool calls involved. And that's the essence of an agent, right? There's often many other subsystems involved. But the essence of an agent is looping over LLM calls with tools. And each of these tool steps outputs a finish reason, which is why did the LLM stop printing text? There's a handful of canonical finish reasons available for all LLMs. They're not always shared, but they're conceptually similar. OpenRouter normalizes them. But one of them is tool call, where it says like, call tool for me, right? It says the LLM is requesting a tool be used. And then when the agent exits, it will finish with efficient reason stop, which is like, I think I, I believe I am done with my task. There is nothing else for me to do here. So we process billions and billions of LLM calls every week. And we monitor and normalize finish reasons across billions and billions of LLM calls. So when we look at the world and look at some major trends, what do we see? Well, over the last year, this is specifically for Anthropic models because I think they're representative in some ways. If you look at in aggregate all models, they might not be, but this is a good representation. This is a percent of requests to Anthropic models on OpenRouter, which I would argue is representative of a lot of global usage trends. How many of those API calls to Anthropic ended with the LLM requesting to use a tool. And the growth here in 1 year is like unbelievable, right? And this is the trend. It is, we see certain models, so open-world models like Minimax M2, which I can talk about in a second, where the number, the tool call rate might be in the 80%, right? It might be really high. But for all Anthropic models, here's the trend. It's like sub-5% to well north of 25%, and this is trending up rapidly. And this is like sort of the exhaust, right? This is the signature, the footprints in the snow of agents being deployed into production. And all the development community and the AI community realizing that like, oh, these can do a lot more than simply answer questions or respond to English text, right? They can start to do tool calls. So we expect to see this trend sort of continue linearly. And for some models, to actually go even higher. And so there's like this quantitative information where we can actually see evidence of agents being deployed into production. This isn't like— and we're talking about real scale here. This is like a large number of requests. This isn't people kicking the tires on agents. The other thing that happened, which is really interesting, probably right around July, which is a bit more anecdotal— I think we have some of the BD and sales folks in this call from the OpenRABBIT side— is like suddenly we started getting questions from customers in calls and in MSAs about our SLAs and our uptime, right? And there was actually this like pretty stark transition where, you know, in March of last year, no one was really asking us about this. We were signing commits, we were signing deals, and, but then all of a sudden in July, people started saying like, well, what's your uptime? Like, how should we think about your uptime relative to the LLM's uptime and like failover and like, how, how should we reason about that? Which is an extremely strong anecdotal, admittedly, but extremely strong indicator that these things have suddenly gone from groups of companies testing them out to being very much in production. And if they go down, it starts to matter. So you all, this group, is now asking us about SLAs when we're negotiating these contracts. So the other major trend playing out along— this is a longer time horizon— but is the number of reasoning tokens. So reasoning models, it's sort of hard to believe, but you'll notice that this is a zero before sort of mid-January of last year. We only just eclipsed the 1-year mark of the release of the first mainstream reasoning model, which was O3 from OpenAI, which was not sort of practical to use in a number of dimensions. It was too expensive and it was slow. But that was really sort of the first reasoning model. And it's actually only been 12 months. It's like kind of wild. There were no reasoning models like 12 months ago. No one was doing it. And now it's 50% of the output tokens that we see are internal reasoning tokens from the models versus like the actual output, sort of the end user or the agentic harness or whatever. So I think a little bit harder for me to predict like how this trends over time. But the shift in one year and the adoption of the reasoning models is like, is pretty stark and dramatic. And what we're seeing as folks are building agents is that they're using a mix of sort of frontier state-of-the-art models, the best stuff that the major labs have put out there, which are in almost all cases reasoning models. They're using those to plan, and then they're often using smaller specialty models to do tool call requests and to execute and look at tools. And so we are seeing that agents that are being deployed today are most frequently a mix of multiple models, or if they may not start off as a mix, they may start with like all sort of a frontier model. And then very quickly folks start managing down cost by bringing open weight models into the mix, particularly around decision-making and tool calling versus like the overall sort of judgment. Where they are less smart from a judgment perspective, but like extremely accurate, extremely good with tool use. And so we're seeing these models get mixed and matched in agentic flows in production in a number of companies. So these reasoning models that are powering the actual like logic, and you have these tool calls happening, a lot of which are happening in small models to actually interact with sort of the outside world in highly accurate, fairly reliable ways that were much less reliable Here we go. All right, I don't see any questions in chat yet, so I'm going to keep going. OK, so one thing that's really interesting about tool calling and inference and models is that— and I think is generally underappreciated— is that not all inference is created equal. And by the way, this slide is— the current state of the world is different than this slide. So don't read this as like Azure is underperforming parasail for GPT-OS-S120B. That is like no longer the case. But this is a true representation of what was going on in the world at one point in time. And so what this is showing here is like a single model, which is GPT-OS-S120B, pretty popular open source model from OpenAI. And this shows against a pretty well-known benchmark, GPQA-16, how did the model perform? Well, in the exact same model, which in theory are the exact same model weights, performed like fairly differently across a number of providers all hosting the same model. And to be clear, like this was done by our friends over at Artificial Analysis. We've partnered with them on a number of things like this. This gap was closed. This was run a few weeks after GPT-OS-S120B was released. And there's a burn-in time for the different providers to learn how to best run these models. But it's worth noting that all inference is not created equal. And for a given model, there's a huge number of different ways that model can actually practically perform in production. And just because it's the same model in the same weights, even in situations, depending on how deep you want to go, like quantizations are the— let's say they're the same. You can still have meaningful differences in how the inference stack is being executed. And how accurate things are. This is just a raw intelligence score benchmark, right? But this can also happen very much, very specifically in tool calling. So this is again not like naming and shaming. This is not categorically true. This changes all the time, right? But this shows for a particular model at a particular period of time how often that model chose to use the tools it was given, right? So this is to say like, hey, you know, is cc@openrouter.ai a valuable customer? You can use Stripe if you want, right? So the exact same model, but hosted by different clouds, decided to use that tool with differing frequency, right? So for this is across like many, many millions of requests, but the model's implementations can subtly vary something really counterintuitive, right? It's very counterintuitive. Like, why would the exact same model with the exact same smarts choose to use tools differently in different situations? And there's a whole bunch of, like, possible reasons for it that don't really come down to the model's judgment, but actually the subtleties of how this stuff is implemented. And so one of the things that we work really hard on at OpenRouter is making sure that agents are successful and that tool calls work. So we get to monitor all this stuff in real time. Now, this is not necessarily a bug. It's not obviously the case that more tool calls are less good than few tool calls. But in general, you would expect the variance in performance to be like a little bit tighter here versus so spread out. So we spent a huge, huge amount of time monitoring this sort of thing and actually have dedicated endpoints for OpenWeight models that only route to providers that we have benchmarked in doing a good job with tool calling. So even amongst tool calling specialty models, we might have 15 different clouds that support a GPT-OSS-120B. But some of those clouds are, tend to be more accurate when calling tools than others for whatever reason. And so we create routing pools of just those endpoints on just those providers to ensure accuracy specifically for Agentic tool callers, right? So there's a whole bunch of stuff that can be done to drive agent accuracy up that is sort of outside of the domain of the actual agent harness and the prompting itself. It's due to the quality of the inference. And so we, we publish a bunch of data around this. We have these things called Exacto endpoints that are tool call specialists. We have things like Nitro endpoints, which are specifically designed to give high throughput, all sorts of like different ways of balancing inference across the needs and price, latency, accuracy requirements of our customers, even when in theory the underlying model is the same. So it's a super interesting and very dynamic ecosystem. And, you know, net-net, as these agents come online, like we've had to do more and more to make sure that we can optimize inference specifically for agents, right? So we can— you can expressively tell OpenRouter, hey, here are my context lengths that I'm expecting or that I actually have. My latency requirements look like this. Do I need to be geo-locked in certain areas? What kind of data policies might I need? You can set routing rules around this and we can basically source the inference that you need based on that set of constraints. And then of course, do things like automatic failover and bursty workloads. This is a big problem for a lot of companies. They have an agent that runs periodically throughout the day with fairly large bursts of token needs. That makes it almost impossible to buy a commit, right? You can't like buy a committed workload for that sort of thing. But because of our scale, because we have so many customers all doing bursty workloads, we can very often, you know, we don't have infinite capacity, but for most intents and purposes, for sort of most companies, um, we can absorb like spiky workloads very easily, um, especially when agents are running on like cron jobs and overnight. We're playing with ideas of like how we can saturate GPUs overnight. So if you have agents that can run not during the day, like, can you get cheaper inference at night? So we're gonna have some experiments running around that. Um, again, I talked about tool calling accuracy, um, and the ability to just simply switch and try new APIs very quickly. Is critical when developing agents. We often see an agent that— a company that's developed an agent. Let's say it's for doing some sort of response to inbound leads. You're like, geez, this thing is just not smart enough. It's not accurate enough to actually put into production. It's close, but it's only right 95% of the time. That's just not good enough. And you might be building on an OpenAI model. And then Gemini 3 comes out in December. And you say, oh, geez, I wonder if this might help. Well, it's non-trivial to switch between OpenAI and Gemini 3 if, like, your company has a relationship with OpenAI or if you have— don't have a Google account spun up. So OpenRouter really helps with agent development in that way as well, because you can immediately try a new frontier model to see if you can get, you know, uh, the, the error rates down enough or the false positive rate down enough so that you can actually release this to production. So, um, I'd say the two big use cases we see for switching models is one, a new frontier model comes out, you're like, Can this now solve my problem? It couldn't before. And two is, I have an agent that's running and it's working, but I'm using frontier models anywhere, everywhere. Can I downclutch to an open weight model now that I understand my use case, get the same results but with much lower cost? Um, so all kinds of things about budget management, guardrails, um, all kinds of things around logging, integration. So basically our, our goal is to be, uh, the best place to source, orchestrate, and control in a way that you feel good about all of the needs of your inference across the company, make sure that all your agents are happy and like getting the tokens they need to run successfully. So that's it for me. Happy to open up to questions.

Hey, Chris, that was really— that was super interesting. Thank you. One question here that I got from the livestream, which I thought was interesting. From your vantage point, hundreds of models and providers. What's the biggest mistake founders make when building agents today?

The biggest mistake founders make when building agents. That's a good question. I mean, I think, I think, you know, look, obviously I'm going to talk my own book a little bit here, right? I think that like building for optionality is sort of the key. I think avoiding lock-in, because I think we would all agree it is extremely hard to predict what we're going to need in 12 months and where that inference will come from and what kind of models we might need. And so what we often— like situations where we often show up where people have built something in a stack that is hard to move to other models or other technologies, and they realize, geez, we really need some more flexibility here. So introduce like us as a gateway layer in order to be able to quickly try out new things, different agents or different models or different clouds without having to like sort of rewrite your code. One of the things that I think is like unintuitive to a lot of people who know OpenRouter from a dev-first standpoint is that like we are most often graduated into, not graduated out of. Like most companies start on with a first-party relationship somewhere, often like burning down cloud credits they've received as part of a startup program. And then realize, oh shoot, I need failover, and then, oh shoot, I need better monitoring, and then you're like, oh no, I need a different model, it's not available on this cloud, right? And that's sort of when we, when we come in. And so I just think knowing that it's gonna, your agent's gonna be unstable, and so how do you plan for the future accurately and make technology choices that preserve optionality against a number of different dimensions?

Okay, yeah, super interesting. Question, how should How should go-to-market teams think about using OpenRouter?

Well, I think, yeah, I mean, one of the go-to-market teams specifically, one of the big kind of open— I don't know if it's an open question exactly. There's sort of, there's obviously going to be a lot of vertically built agents that are specialized SaaS, right? There may be people on here. Certainly there's a lot of people on here who are buying that stuff. There may be a lot of people on here who are also building those things, right? But you're like, Hey, Harvey is like the agentic thing for lawyers, right? And you have, you know, agentic doctors and agentic, you know, customer support. And there's all these different verticals. So I think in, for go-to-market teams, there's going to be, there are, there are going to be, and they're going to emerge some winners who are really great verticalized go-to-market agentic stuff that's like packaged up and really good wrappers. But the count, the sort of counterweight to that is that It's obviously one of the other major trends here that's like way easier to write custom software these days. Right. And it's actually nearly trivial to, in some cases, write your own in-house agents. So the question is like, how will the market sort of settle? And is it the case that like, look, a 4-person GTM team who has some tech jobs but not engineers can just like kind of build the agent they need through Claude code It's like, okay, here, I need an agent to do this thing. I'm going to use a framework like Crew or Maastra or AWS Agent Core or whatever, or just roll it from scratch and do it yourself. So I think there's a stronger argument these days, especially in go-to-market stuff, to maybe try rolling it yourself and see how far you get, because those tools are just becoming incredibly powerful. And so I think just being unafraid to like just build your own agent internally using an open source framework or using a hosted framework and seeing how far you get before going to sort of like bundled SaaS stuff. Because go-to-market is really nuanced that like in many ways is the way you touch your customers and it is the identity of the company. It is a product in its own right. And so merits a lot of sort of custom thought and ability to have full control over it. Very cool.

There's a question here in the chat. I don't know if you see it about NVIDIA and Nemo.

Yeah. We do host the Nemo models. If you just go to openrouter.ai/models, you can see what we host. Something interesting about the Nemo models, I think, you're going to have to double check this because if I'm wrong, this would be a bad thing to be wrong about. Uh, the Nemo models are interesting because they— this is like a wonky AI thing, but they, um, allow distillation, which is kind of interesting. Distillation is when you like pull a bunch of information out of a model through a huge series of prompts and then train a different model on those outputs. That is like very much not allowed for almost all models. Um, but interestingly, uh, NVIDIA, obviously they want more people to build more models, so they've built Nemo and Neutron and allow people to distill them to build their own in-house models, which is like a kind of interesting use case around those specifically.

Okay, very cool.

What do we got here?

Hold on. Oh, got a thank you from Michael.

Yeah.

What do you think founders should optimize for first? Model quality, inference routing, or observability?

None of those. I mean, look, the key to getting an agent really work, it's like, look, go, like, use OpenRAWR. Like, if you're starting from scratch, my advice would be use OpenRAWR, pick a frontier state-of-the-art model. So you're like, if it's going to fail, it's not because you're like, the model's not smart enough, you should pick something smarter, right? And then, you know, the, you're going to need some good traceability understanding of like where agents are falling down. But then largely, this is still a lot of the magic is in the context engineering and getting the right information that this thing needs in the context of the prompts. So I would think about the routing and the model piece as something that's like, look, get the best stuff in market from a routing standpoint. And then from a model standpoint, start development there if you're coming to this new. And basically take those factors out of play as a reason to fail. That's not a good reason to fail. And then given the best quality inference and the best quality routing, the best quality model you can, see if your use case is possible and see how well you can do and build from there. But if you do a great job building and you do use like crappy quality inference and you draw a false negative, right, this could have worked, but it didn't for sort of more mechanical reasons than is really necessary.

Yeah, that's great advice. I think we're just about at time here, Chris. If— actually, there's one last question if you want to address it in the chat.

I think it's a couple of things. I mean, it really varies. It depends on like, it depends on the way the enterprise is managed. Very often there's like a lot of, there's a lot of just like, in large enterprises, obviously, red tape and just getting to the inference quality and permissions and throughput and bandwidth and budget that you need to actually run this stuff. The other really big one, it's a very legitimate concern, is like what's going into this stuff and where, even if you're working with like pretty good, I'll wrap in 30 seconds, I promise, pretty good, reputable inference providers. Like, where are their data centers? Do they actually own the GPUs, or do they have like GPUs that are leased in a number of data centers in different places? And where's the decryption happening? And like, that data policy stuff really matters, especially like enterprises handling sensitive data. So, um, figuring out how to get a good control environment over where your prompts are going is critical. And we've made huge investments in being able to do that, and there's like a ton more to go there. But I think in terms of broad rollout, it's often the case that like, this is sometimes the most sensitive data in your company, and you're sort of just like It's going to AI, right? And it's like making companies comfortable with exactly what's happening and that that environment is controlled is like, is really, really important. Yeah.

Yeah, that's great. Well, Chris, thank you so much for a great session. If anyone, and your email for anyone who wants to reach you?

cc@openrouter.ai.

Okay, perfect. cc@openrouter.ai. And thank you all for joining today's AI Day session. I think there's one more session on tap. And Chris, thanks so much for great content.

You got it. Thanks.

All right, bye, bro.