OpenRouter: The Only AI Tool You'll Ever Need | Founder Alex Atallah

What if I told you there was a single website you could go to where you can chat to any major AI model from one single interface? It's kind of like ChatGPT, but instead every prompt gets routed to the exact AI model that will do the best job for whatever your prompt might be. Well, on today's episode, we're joined by Alex Atallah, the founder and CEO of OpenRouter AI. It's the fastest growing AI model marketplace with access to over 400 LLMs, making it the only place that really knows how people use AI models and more importantly, how they might use them in the future. It's at the intersection of every single prompt that anyone writes and every model that they might ever be. Alex Atallah, welcome to the show. How are you, man?

Thanks, guys. Great. Thanks so much for having me on.

So it is a Monday. Um, how does the founder of OpenRouter spend his weekend? Presumably. You know, out and about, chilling, relaxing, not at all focused on the company.

Oh, I usually, I love weekends with no meetings planned and I just go to a coffee shop and just have tons of hours stacked in a row to do things that require a lot of momentum buildup. So I did that at coffee shops on Saturday and Sunday, and then I watched Blade Runner again.

I can't.

Okay.

Um, well, so when we were preparing for this episode, Alex, um, I couldn't help but think that you've had a pretty insane decade of startup foundership, right? Um, so OpenRouter is kind of like your second major thing that you've done, but prior to doing that, you were the founder and CTO of OpenSea. the biggest NFT marketplace out there. And now you're focused on one of the biggest AI companies out there. So it sounds like, you know, you're, you're at kind of like the pivot point of, uh, two of the most important technology sectors over the last decade. Um, how do you, can you just give us a bit of background as to, you know, how you ended up here and more importantly, where you started? Walk us through the journey of OpenSea and, and how you ended up at OpenRouter AI.

Yeah. So. I co-founded OpenSea with Devin Finzer at the very beginning of 2018, very end of 2017. It was the first NFT marketplace and it was not dissimilar to OpenRouter in that there was a really fragmented ecosystem of NFT metadata and like media that gets attached to these tokens. And it was the first example of something in crypto that like could, could be non-fungible, meaning it's a single thing that can be traded from person to person. Most things in the world are non-fungible, like a chair is non-fungible, a currency is fungible. So it was back, back in 2018, no one was really thinking about crypto in terms of non-fungible goods. And the problem with the non-fungible goods is that there weren't any real standards set up. There was a lot of heterogeneous implementations for how to get a non-fungible item represented and tradable in a decentralized way. So OpenSea organized this very heterogeneous inventory and put it together in one place. And we came up with a metadata standard. We did a lot of, like, a lot of work to really make the experience super good for each collection. And you see a lot of those, a lot of similarities with how AI works today too, where there's also just a very heterogeneous ecosystem, a lot of different APIs and different, like, features supported by language model providers and OpenRouter. Similarly does a lot of work to organize it all. Um, I was at OpenSea, uh, until 2022, um, when I was kind of feeling the itch to do something new. And, um, and at the very end of— I left in August, and then ChatGPT came out a few months later. And, uh, and my biggest question around that time was whether it was going to be a winner-take-all market because OpenAI was very far ahead of everybody else. And, um, you know, we had Cohere Command, we had a couple open source models, um, but OpenAI was the only really usable one. I was doing little projects to experiment with the GPT-3 API, and, uh, and then Llama came out in January. Um, really exciting. About a tenth the size, won on a couple benchmarks, but it wasn't really chatable yet. And, uh, and it wasn't until, uh, a few months later that somebody, a team at Stanford, distilled it into a new model called Alpaca. Distillation means you, you take the model and you customize it or fine-tune it on a set of synthetic data that they made using ChatGPT. As a research project. And, uh, and that was the, it was the first successful major distillation that I'm aware of. Um, and it was an actually usable model. I was like on the airplane talking to it. I was like, wow, this is, if it only took $600 to make something like this, then you don't need $10 million to make a model. There might be like tens of thousands, hundreds of thousands of models in the future. And suddenly this started to look like a new economic primitive, a new building block that people would, that, that kind of deserve their own place on the internet. And it did. There wasn't one. There wasn't a place where you could discover new language models and see who uses them and why. And that's how OpenRouter got started.

That's amazing. So one of the things that we're, we're obsessed with on this channel in particular is, is exploring frontiers and how to properly see these frontiers and analyze them and understand when they're gonna happen. And when I was going through your history, you, you have this talent consistently over time. And I, even as far back as early on, I read you, you were hacking Wi-Fi routers in a hackathon. You were very early to that. You were early to the NFTs. You were early to understanding AI and the impact that it would have. And what I'd love for you to explain is, is the thought process and the indicators you look for when exploring these new frontiers, because clearly there's some sort of pattern matching going on. Clearly you have some sort of awareness of, of what will be important and why it will be important, and then inserting yourself into that narrative. So are there patterns? Are there certain things that you look for when searching for these new opportunities and that led you to make these decisions that you have?

I think there's, there's a lot to be said for finding enthusiast communities and, and seeing if you're going to join it. Like, can you be an enthusiast with them? Um, like whenever something new comes out that has like some kind of ecosystem potential, there's, there are going to be enthusiast communities that pop up and the internet has made it self-serve. You could just join the communities. Discord, I think, is an incredible and super underrated platform because the communities feel kind of private. You're like getting, you don't feel like you're, you know, seeing somebody trying to get, you know, like advertise something for SEO juice. There's no SEO juice in Discord. Um, it's, it's just people talking about what they're passionate about and, and it goes, it gets really niche. Um, and when you find like, like an interest group in Discord that like has to do with some, some new piece of technology that's just being developed right now and doesn't really work very well at all. Um, you get people who are just trying to figure out what to do with it and how to make it better. And I think that's like, that's the first core piece of magic that jumps to mind. You got, there's gotta be like a willingness to be weird. Cause like, if you jump into any of these communities at face value, it's stupid.

Yeah.

It's like, oh, this is like just a game. Or it's like a really weird game. I mean, I'm not really into that kind of collectible game. So I'm going to leave right now. And you, and not only do you have to be aware, but you have to be creative. Like, okay, it's, you know, this is, these are just cats on the blockchain and people are just like trading cats back and forth. It's, you can't like look at the community as simply that. Like think about what you could do with it. Like what does this unlock that wasn't achievable before? And I think there are people who just are good, who will do this and they'll join the communities and brainstorm live and you can see everybody brainstorming in real time. But like another incredible example of this was the Midjourney Discord. You know, it became the biggest server in Discord by far. And why did that happen? Well, it started with something weird, silly, maybe not super useful, but you could see all the enthusiasts remixing and brainstorming live how to turn it into something beautiful and how to make it useful. And then just exploded. I, it's the most, it's the, it's the most incredible like niche community, uh, I think that Discord has ever seen because of like how useless it started and how insanely exciting it became. So, um, like, I mean, I, I think I saw Big Sleep. I was like playing around with this model called Big Sleep in 2021. That, uh, let you generate images that look kind of like DeviantArt. Okay. And, uh, and you could see— you could like— they're all animated images and they— none of them really made sense, but you could get some really cool stuff. Not like potentially something you'd want to make your desktop wallpaper. And if you're really like deep in some DeviantArt communities, you know, you kind of appreciate it. And so that, that was like, oh, this is like a kernel of something here. And it took like another year or two before Midjourney started to like pick up.

Sure.

But that was like—

Where were you seeing all of this, Alex? Like, where were you scouring? Just random forums or just wherever your nose told you to go?

But basically there's this Twitter account, I'm trying to remember what it's called, that posts AI research papers. Mm-hmm. And and like kind of tries to show what you can do with them. And I discovered this Twitter account in like 2021. And I think it was not, it was, it wasn't at all like related to crypto, but it was a way, you know, Big Sleep was like the first thing I saw that used AI to generate things that could potentially be NFTs. So I started experimenting around like how, how much you could direct it to make an NFT collection that would make any sense. It was very, very difficult. But that was how, that was like the first generative.

And this was before you were even thinking about starting OpenRatter, right?

Yeah, yeah, this was back, this was when I was full-time at OpenSea. Oh, is, yeah, I got the, it's Acolic. This Twitter account. Acolic, okay, all right. Highly recommend it. They basically post papers and like explainate and explore how this paper gets useful. They post animations. Like they make it, they make AI research like kind of fun to engage with. And that was, that was my like first experience.

Okay. So, I mean, that's a massive win for X or formerly as it was known back then, Twitter as a platform, right? It gave birth to kind of like two of the biggest technologies, crypto, also known as crypto Twitter. Um, and now apparently like, you know, all the AI research stuff, which kind of put you on to the path that led you to OpenRata. So if I've got this right, Alex, you know, you were full-time at OpenSea, which, you know, multi-billion dollar company, loads of important stuff to do there, but you still found the time to kind of scour this fringe technology because that's what AI was at the time prior to kind of GPT-2 or GPT-3. No one really knew about this. And you were playing around with these gen AI models, these generative AI models that would you know, create this magical little substance and maybe it came in the form of a picture or a weird little cat and you kind of like jumped into these niche forums of enthusiasts, as you say, and kind of explored that further. And it sounds like you kind of like honed that even beyond your journey from OpenSea when you left. I remember actually meeting you in this kind of like this abyss between you leaving OpenSea and starting OpenRouter. Where you were kind of brainstorming a bunch of these ideas. And I remember a snippet from our conversation in like one of the WeWorks here, uh, where you just kind of like had whiteboarded a bunch of AI stuff. And, and one of those things was kind of like the whole topic of inference. And if I'm being honest with you, Alex, I had no idea what that word even meant back then. I was, I was extremely focused on, on all the NFT stuff and all the crypto stuff. My background's in in all of that, but, um, I, I just found that fascinating that you always had your nose in some of the, uh, early communities. And I think that's a really important lesson there. Um, I want to pick up on something that you actually brought up when you said you discovered kind of like your path to OpenRouter, um, Alex Atallah. And that is, you said you were playing around with these early AI models. So not the GPTs, before Claude was even created. You're playing around with these random models that you would find either on forums, on Twitter, or on Reddit, right? And you would experiment with them. And I find it fascinating that back then, even when GPT became a thing, you were convinced that there would be hundreds of thousands— what did you say? Hundreds of thousands of AI models? Back then, that wasn't a normal view. Back then, everyone was like, you need hundreds of millions of dollars. Maybe it was tens of millions of dollars back then, and it was going to be a rich man's game.

Yeah, it was basically the Alpaca project that, um, that kind of put me over the fence on model— on, on there being like many, many, many models instead of just a very small number.

And can you explain what the Alpaca project is for the, for the audience?

Yeah. So, um, the Alpaca project But, you know, uh, after Llama came out, it— you really could not chat with it very well. It was, it was a text completion model. Um, they— there were like a couple benchmarks where it beat GPT-3, and it was about a tenth the size of what, what most people thought GPT-3 was, was sized at. Um, so it was a pretty incredible achievement, but You can, it wasn't really like, it didn't, the user experience wasn't there. And, um, the Alpaca project took ChatGPT and generated a bunch of synthetic outputs. Um, and then they fine-tuned Llama on those synthetic outputs. And this did two things to Llama. It taught it style and it taught it knowledge. It taught it like the style is like how to chat, which was the big user experience gap. And, uh, and it made it smarter. It, it, you know, like you can fine-tuning transfer to both style and knowledge and the model would like respond to things that it had, you know, like what the content of the synthetic data, like was reflected in the model's performance on benchmarks after that point. So, um, so if you can do that without revealing all the data that goes in, Now there's like a way you could sell data via API without like just dumping all the data out to the world and then never being able to monetize it again. So there's like a brand new business model around data that emerges. You get like the ability to create, to like work towards open intelligence and build like new architectures, test them more quickly, and, and, uh, uh, fine-tune them quickly. Basically, you can build on top of the work of giants. I mean, you don't have to start from zero every time. A lot of like the biggest developer experience innovations just involve like giving developers a higher stair to start walking up so they don't have to start at the bottom of the staircase every single time. Um, and, you know, that was like the, the big, like, generous give that Llama had for the community. Um, and it wasn't, you know, it wasn't the only company doing open source models. Mistral, um, came out with 7B Instruct, uh, a few months later. It was an incredible model, you know. Then they came out with the first open Wait's Mixture of Experts a few months later, you know, it just, it felt like actual intelligence but completely open. And all of these provide like higher and higher stairs for other developers to kind of like, you know, basically to crowdsource new ideas from the whole planet, uh, and, and let these new ideas build on top of really good foundations. So And, and you know, when that, when that like whole picture started to form into place, it felt like, okay, this is gonna be like a, you know, huge inventory situation. You kind of like NFT collections were a huge inventory situation, obviously completely different, really different market dynamics, really different type of, of goal that buyers have. And so a lot of like my early experimentation, like I made like a, Chrome extension called Window AI. I did like a few other things were just about learning how the ecosystem works and like what makes it different and how the, like, like what people really want, what developers really want.

So that leads us to OpenRouter itself, right? So I kind of want you to help explain to the listeners who aren't familiar with OpenRouter what it does. Uh, cause I think a lot of people there, the way they interact with an AI is they send a prompt to their. Model of choice. They use ChatGPT or they use the Groq app or they're on Gemini and they kind of live in these siloed worlds. And then the next step up from the people are those kind of who use it professionally, who are developers. They're interacting with APIs. Maybe they're, they're not interfacing with the actual UI, but they're, they're calling a single model and OpenRouter kind of exists on top of this, right? Can you walk us through how it works and why so many people love using OpenRouter?

OpenRouter is an aggregator and marketplace for large language models. You can kind of think of it as like a You know, like Stripe meets Cloudflare for both of them. It's like a single pane of glass. You can orchestrate, discover, and optimize all of your intelligence needs in one place. You know, one, one billing provider gets you all the models. There's like 470+ now. Like all the models, like they sort of implement features, but they do it differently. And they also, there's a lot of like intelligence brownouts as Andrej Karpathy calls them.

Yeah.

Where models just go down all the time. Even the, you know, even the top models like Anthropic and Gemini and OpenAI. So what we do is, you know, we like, developers need a lot of choice. CTOs need a lot of reliability. CFOs need predictable costs. Uh, CISOs need like complex policy controls. All of these are inputs to what we do, which is build like a single pane of glass that makes models more reliable, lower cost, um, gives you more choice, and, uh, and then, and helps you choose, uh, between all the options for where to source your intelligence.

How does it work? Uh, because I would imagine, like, Ijaz and I on the show, we frequently talk about benchmarks, right? Where a certain model is the best at coding. And that infers that maybe you should go to that model to do all of your coding needs because it's the best at it. But it would appear as if it's not true if you're routing through a lot of different providers. So how do you consider which provider gets routed to when and how to get the best result for what you're asking?

So we've taken a different approach so far, which is instead of like focusing on a production router that picks the model for you. Mm-hmm. We try to help you choose the model. So we build lots of— we create lots of analytics both on your account and, and on our rankings page to help you browse and discover the models that, like, the power users are really using successfully on a certain type of workload. Because we think, like, developers today primarily want to choose the model themselves. Switching between model families can result in like a lot, like very unpredictable behavior. But once you've chosen your model, we try to help developers not need to think about the provider. There are like sometimes dozens of providers for a given model, all kinds of companies, including the hyperscalers like AWS, Google Vertex, and Azure, um, and, uh, like scaling startups like Together, Fireworks, Deepinfra, um, and a long tail of providers that provide like very unique features, very like exceptional performance. Um, there's all kinds of differentiators for them. So what we do is we, we collect them all in one place, and if you, if you want a feature, You just get the providers that support it. If you want performance, you get prioritized to the providers that have high performance. If you really are cost sensitive, you could prioritize to the providers that are really low cost today. And, uh, and we, we basically create all these lanes. There's like innumerable ways you could get routed, but you're in full control of the, of the overall user experience that you're aiming for. And that's what, that's what we found that was missing from the whole ecosystem was just a way of doing that. And, and we, you know, we get like between on average 5 to 10% uptime boosts over going to providers directly just by load balancing and, and sending you to the top provider that's like up and able to handle your request. And we do in like a, we really focus hard on efficiency and performance. We only add about like 20 to 25 milliseconds of latency on top of your request, and it all gets deployed very close to your servers up the edge. So we overall get, you know, just we stack providers, we figure out like what you can benefit from that everybody else is doing and, and just give you the power of big data as a, as a developer, just accessing your model of choice.

So it kind of allows you to harness the collective knowledge of everybody, right? You get all of the data, you have all of the queries, you know which yields the best result, and you're able to deliver the best product for them. Now, in terms of actual LLMs, Ijaz actually pulled this up, uh, just before, which is a leaderboard. And I'm interested in how you guys think about LLMs, which are the best, how to benchmark them, and how you route people through them. Is there a specific— do you believe that benchmarks are accurate and do you reflect those in the way that you route traffic through these models?

In general, we have taken the stance that we want to be the capitalist benchmark for models. Like, what is actually happening? And, uh, and part of this is that I really think both the law of large numbers and the enthusiasm of power users are really, really valuable for everybody else. Like when you are routing to, um, like Claude in, uh, let's say you're routing to Claude 4 and you're based in Europe. Um, there are, you know, all of a sudden there might be like a huge variance in throughput from one of the providers. And you're only able to detect that if like some other users have discovered it before you. And so we route around the provider that's like running kind of slow in Europe and send you, if your data policies allow it, to a much faster provider somewhere else. And that allows you to get faster performance. So like, um, that's like on the provider level how like numbers help. On the like model selection level, like what you see on this rankings page here, um, power users will— like when we put up a model, like we put up a, um, a new model today from a new model lab called Z AI, like, like the power users instantly discover it. We have this LLM enthusiast community that, uh, that dives in and, and really figures out like what a model is good for and along like a bunch of core use cases. The power users figure out which workloads are, are interesting. And then you can just see in the data, uh, what, what they're doing and everybody can benefit from it. That's why we like open up our data and share it for free on the rankings page here.

I'm seeing this one consistent unit across all these rankings, Alex, which is tokens, right? And Josh and I have spoken about this on the show before, but I'm wondering how, like, you've chosen this specific unit to measure, you know, how good or effective these models are or how consumed or used they are. Can you tell us a bit more as to why you've picked this particular unit and what that tells you as like the OpenRider platform as to like how are users using a particular model?

Yeah, I think, I mean, I think dollars is a good metric too. The reason we chose tokens is primarily because we were seeing prices come down really quickly for most of the— OpenRouter has been around since the beginning of 2023, and, and I didn't want a model to be penalized in the rankings just because, like, the prices are going down really dramatically. Now, like, there's a paradox called Jevons' paradox, which is that when prices decrease, like, 10x, users' use of some, like, component of infrastructure increases by more than 10x. And so, maybe they wouldn't get any likes at all. But I thought there were, there were some other advantages to using tokens too. Tokens like don't have this penalty and don't, don't rely on Jevons paradox, which can have like a lot of lag. They also, they also are a little bit of a proxy for time. You know, a model that is like generating a lot of tokens and doing so for a while across a lot of users means that a lot of people are like reading those tokens and actually doing something with them. And same goes for input. Like, if I, if I really want to, like, send an enormous number of documents and the model has, like, a really, really, really tiny prompt pricing, I think that's still valuable and something that we want to see. We want to see that this model is, like, processing an enormous number of documents. Um, that's, like, a use case that should show up in the rankings. Um, and so we decided to go with tokens. We might, like, add dollars in the future. But, uh, I think tokens are, um, you know, they, they don't have this like Jevons paradox lag. Um, and, and there wasn't anything else, like nobody was doing any kind of like overall analytics. We didn't see any other company even do it until Google did a few months ago where they started publishing the total amount of tokens processed by Gemini. So we'll see like what the, you know, which use cases really need dollars, but tokens have been holding up pretty well. Yeah.

I mean, this dashboard is awesome. And I recommend anyone that's listening to this that can't see our screen to get on OpenRouter's website and check it out. I've been following it for the last 2 weeks, kind of pretty rigorously, Alex, and What I love is you can literally see, um, so 2 weeks ago Grok 4 got released, right? And Josh and I were making a ton of videos on this and we were using it with pretty much everything that we could do. Um, and then this other model came out of China pretty much a few days after called KIMI-K2. And I was like, oh yeah, whatever. This is just some random Chinese model. I'm not going to focus on it. Um, and then I kept seeing it in my feed and I thought, okay, maybe I'll give this a go. And I kind of like went straight to OpenRouter to just kind of like almost gauge the interest from a wider set of AI users. And I saw that it was skyrocketing, right? And then I saw that, you know, Qwen dropped their models last week. And again, I came to OpenRouter and it like preceded the trend, right? People had already started using it. So I love how you describe OpenRouter as this kind of like prophetic orb, basically. Where the enthusiasts and the community itself can kind of like front run very popular trends. And I think that's a very powerful moat and kind of on this path, Alex, I noticed that, um, a lot of these major model providers see the value in this, right? So, uh, if I'm not mistaken, OpenAI kind of like used your platform to kind of secretly launch their Frontier model before they officially launched it. Right. Can you walk us through, um, you know, how that comes about and more importantly, why they want to do that and why they chose OpenRadar to do that?

Uh, OpenAI will sometimes give, uh, early access to their models to some of their customers for testing. And, uh, we asked them if they wanted to try a stealth model with us, which we had never done before. Um, it involved like launching it as under another name and seeing how users respond to it without having any bias or sort of inclination for or against the model, um, at the onset. And it would be like a new, a new way of testing it and a new way of— it was like an experiment for both us and them. And they generously, uh, decided to take, you know, the leap of faith and try it. And, uh, we launched GPT-4.1 with them, uh, and we called it Quasar Alpha. Yep. And it was a million, uh, token context length model, opening its first very, very long context model. And it was also optimized for coding. And the incredible— there were a couple incredible things that happened. First, we have this community, uh, of benchmarkers that run open source benchmarks, and we give a lot of them grants to help fund the benchmarks, grants of OpenRouter tokens. They'll just run the suite of tests against all the models, and some of them are very creative. Like there's one that tests, uh, like the ability to generate fiction. There's one that tests, um, like how— like whether it can make a 3D object in Minecraft called MC Bench. Um, there are a few that test different types of coding proficiency. Um, there's one that just focuses on how good it is at Ruby, because Ruby is— turns out a lot of the models are not great at Ruby. There are a lot of like languages that all the models are pretty bad at. And, uh, and so we have this like long tail of very niche benchmarks, and all the benchmarkers ran, you know, like for free, their benchmarks on Quasar Alpha and found pretty incredible results for most of them. Um, and so the model got like, you know, OpenAI got, got this feedback in real time. We kind of like, um, helped them, uh, find it. And, and they, like, uh, they, they made another snapshot which we launched as Optimus Alpha. So then they could compare the feedback that they got from the two snapshots. Um, and, and then they— and then like 2 weeks later they launched GPT-4.1 live for everybody. So it was like a, uh, was it an experiment for us? And, and we've done it again since, uh, with another model provider that, uh, that's still working on it. Um, and it's kind of like a cool way of learning, of like crowdsourcing, uh, benchmarks that you wouldn't have expected, and also getting unbiased community sentiment.

That's great.

So now when we see a new model pop up and, um, we want to test GPT-5, we know where to come to to try it early. We'll see, because rumor is it's coming soon. So we'll be, we're on your watch list. Um, but having, I do want to ask you about open source versus closed source, because this has been an important thing for us. We talk about this a lot. You have a ton of data on this. I'm looking at the leaderboards there. There are open source models that are doing very well. Closed source. What are your takes in general? How do you feel about open source versus closed source, um, models, particularly around how you serve them to users?

Both models, both types of models have supply problems, but the supply problems are very different. Typically what we see with closed source models is that there's, there's very few suppliers, um, usually just one or two. Um, like with Groq, for example, there's Groq Direct and there's Azure. Um, with Anthropic, there's Anthropic Direct, there's Google Vertex, there's AWS Bedrock. Um, and then we also like deploy it in different regions. Like we have an EU deployment. For customers who'd like only want their data like to stay in the EU. And, uh, and we do custom deployments for the, for the closed source models too, to just kind of guarantee good throughput, high and high rate limits for people. Um, but, uh, uh, the like a tricky part is that like the, the demand— usually the, like, the closed-source models are doing most of the tokens on OpenRouter. Um, it's, it's dominant, you know, it's probably 80-ish, 70 to 80% closed-source tokens today. Um, but the, the open-source models, um, have a much more fragmented supply, like, sell-side order book.

Yeah.

Um, and, and like the rate limits for each provider is like a, like less stable on average. Um, it usually takes a while for the hyperscalers to serve a new closed source, a new open source model. Um, so we, so the load balancing work that we do on, um, open source models tends to be a lot more valuable. The load balancing work that we do for closed source models, uh, tends to be very focused on like caching feature awareness, making sure you're getting like clean cache hits and only transitioning over to new providers when your cache is expired. Um, for open source models, like there's way less caching, like very, very few open source models implement caching. Uh, and so like switching between providers becomes more common. And, uh, like we, we also track a lot of quality differences between the, the open source providers. Some of them will deploy at lower quantization levels, which means like it's kind of like a way of compressing the model. Um, generally doesn't have an impact on the quality of the output. Uh, but, and yet we still see some odd things from some of the open source providers. And, and so we run tests internally to detect those outputs, and we're building up a lot more muscle here soon. so that like they get pulled out of the routing lane and, and don't affect anyone.

So closed source accounts for 80% or something like that, a very large amount. Do you see that changing? Because that, that post we just had, it said 9 out of the 10 fastest growing LLMs last week, they were open source. And every time it seems like China comes out with another model, it was Kimmie K2 a week or two ago, um, it's, it kind of really pushes the frontier of open source forward and the rate of acceleration of open source seems to be as fast, if not faster than closed source, where it's just, it's making these improvements very quickly. It has the benefit of being able to compound in speed based because it's open source and everyone can contribute. Do you think that starts to change where the percentage of tokens you're issuing are from open source models versus closed source? Or do you continue to see a trend where it's going to be Google, it's going to be OpenAI that are serving a majority of these tokens to users?

In the short term, we're likely to see open source models continue to dominate the fastest growing model category on OpenRouter. And the reason for that is that, um, a lot of users who come for a closed source model but then decide they want to optimize later, either they want to like, you know, save on costs or, uh, or like try out a new model that's supposed to be a little bit better in some direction that their, their app cares about or their use case cares about. Um, and then they, they leave the closed source model and go to an open source model. So open source tends to be like a last mile optimization thing. Yeah. Making a big generalization here because, you know, the reverse can happen too. Um, and so because it's a last mile optimization thing, the jump from this model is not being used at all to this model is really being used by a couple people who like have left Claude 4 and, uh, and want to try like some new coding use case will be like bigger, um, than, you know, the open— the closed source models which start at a really high base and don't have like growth quite as dramatic. So, um, The other part of your question though was whether there's going to be like a flipping of closed or some sort of like chipping it away at that monopoly of closed source tokens. It's hard to predict these things because, you know, I think like the, the biggest problem today with open source models, um, is that the incentives are not as strong. Like the model lab and the model provider They've, you know, they're sort of established incentives for how to grow as a company and attract good, high-quality AI talent. And giving the model weights away impairs those incentives. Now, like, we might see, you know, this is where we might see like decentralized providers. Helping in the future. A way for like, you know, like a really good incentive scheme that like allows high-quality talent to work on an open-source model that remains open weights at least, like could fix this. I like, you know, I stay pretty, I try to stay close to the decentralized providers. And like learn a lot from them. There's some like cool on the provider side on like on running inference. I think there's some really cool incentive schemes being worked on, but on actually like developing the models themselves, I haven't seen too much, unfortunately. So I think if we see one opening in the radar and until we do, I personally doubt it. TBD.

Do you have personal takes on how you feel about open source versus closed source? Because this has been a huge topic we've been debating too, is just the, the ethical concerns around alignment and closed source models versus open source. When you look at the competitors, China, generally speaking, is associated with open source, whereas the United States is generally associated with closed source. And we saw Llama, um, and, uh, Meta, like, release the open source models, but now they're raising a ton of money to pay a lot of employees a lot of money to probably develop a closed source model. So it seems like the trends are kind of split between U.S. and China. And I'm curious if you have any personal takes, even outside of OpenRouter, of, of which you think serves better for the long-term outlook on, I mean, the position of the United States or just the general safety and alignment conversation around AI?

I mean, like, a, a very simple fundamental difference between the two is that, um, an innovation in open-source models can be copied more quickly than an innovation in closed-source models. So In terms of velocity and how far ahead one is over the other, that is a massive structural difference that means that closed source models should be theoretically always ahead until a really interesting incentive scheme develops, like I mentioned before. I think, and I think that's, You know, I don't see like evidence that that's going to change. In terms of China versus the US, it's, I think it's very interesting that China has not had like a major closed source model. And I don't really see a great reason why, like I'm not aware of any reasons that's not going to be the case in the future. My prediction is that there's going to be a closed-source model from China. And if, it's possible that DeepSeas and Moonshot and a few of, and Quen have built up really sticky talent pools, but generally with talent pools, after enough years have passed, people quit and go and create new companies and build new talent pools. And so we should see some of that. Like, it's not the case that the AI space has NDAs or non-competes that the hedge fund space has. Like, that might happen in the future too, but assuming that, you know, the current non-compete culture continues, um, we should have, there should be like more companies that pop up in China over, over time. And I'm betting that some of them will be closed source. And my guess is that like the two nations will start to look more similar.

Yeah, I guess, um, you know, that's why you have Zuck dishing out $300 mil to a billion dollar salary offers to a bunch of these guys, right? One more question on China versus the US. I, I kind of agree with you. I, didn't really expect China to be the one to lead open source anything, let alone the most important technology of our time. Do you think is there a secret sauce to building these models, Alex Atallah? And I know this might be out of the forte of OpenRaja specifically, but as someone who has studied this technology for a while now, I'm struggling to figure out, you know, what advantage they had, you know, they're discovering all these new techniques and maybe the simple answer is like constraints, right? They don't have access to all of NVIDIA's chips. They don't have access to infinite compute. So then maybe they're forced to kind of like figure out other ways around the same kinds of problems that Western companies are focused on. Um, but it's pretty clear that America with all its funding hasn't been able to, uh, make these frontier breakthroughs. So I'm curious whether you are aware of or know some kind of technical moat that Chinese AI researchers or these AI teams that are featuring on OpenRata day in and day out have over the U. S.

Well, um, I don't know. There, like, there are certainly some that they've come up with. That, um, like, like DeepSeq had a lot of very cool inference innovations that they published in their paper, but a lot of what they published in the original R1 paper were things that looked like that OpenAI had done independently, uh, themselves. So, uh, I, like, on the inference side and on, uh, some of the model side, I think, like, DeepSeq, we, we we had talked to their team for years before R1 came out. They had many models before that, and they were always like a pretty sharp, like, team for doing inference. Um, like, they came up with like the best user experience for caching prompts long before Deep Super R1 came out, and they had very good pricing. Um, they, uh, they were just— they were like, you know, by far the strongest Chinese team, um, that we were aware of, uh, well before that happened. And so I'm guessing there was like some talent, uh, accumulation that they were working on in China for people who wanted to stay in China. And yeah, that's, that's a huge advantage. Like American companies are obviously not doing that. There's a little— Zuck is very on point that a lot of this is just based on talent. Um, there are a lot of AI is open and out there and just like, and very composable. It's like a big tree of knowledge. There's a paper that comes out and it cites like 20 other papers and you can go and read all of the cited papers and then you like have kind of a basis for understanding the paper, but you really have to go one level deeper and read all the cited papers two levels down to really understand what's going on. And it's just that no, very few people can do that. And it takes like a lot of years of experience to like actually apply that knowledge and learn all these things that have not been written in any paper at all. And there's just such a small number of people who can really lead research on all the different dimensions that go on to making a model. And like the border between China and the US is pretty defined. You have to leave China and move to the US and really establish yourself here. So I do think there's like country arbitrage. There's like, there's, you know, the hedge fund background arbitrage. There's hardware arbitrage. Like there's like a ton of hardware that's only available in China, but not here, vice versa. That creates an opportunity. And this, this will just continue to happen.

Yeah, I, I think this arbitrage is, is fascinating. I read somewhere that there's like probably less than 200 or 250, uh, researchers in the world that are kind of like worthy of working at some of these frontier AI model labs. And I looked into some of the backgrounds of the team behind KIMI-K2, which is this recent open source model out of China, which kind of like broke all these, um, crazy rankings. I think it was like a trillion parameter model or something crazy like that. And a lot of them worked at some of the top American tech companies. Um, and they all graduated from this one university in China. I think it's Tsinghua, uh, which apparently is like, you know, the Harvard, uh, of AI in China. Right. So, so pretty crazy. Um, but Alex, I wanted to shift the focus of the conversation to a point that you brought up earlier in this episode, which is around data. Okay. So here's the context that like Josh and I have spoken about this at length, right? We are obsessed with this feature on OpenAI, which is memory, right? And I know a lot of the other memory, sorry, a lot of the other AI models have memory as well. But the reason why we love it so much is I feel like the model knows me, Alex. I feel like it knows everything about me. It can personally curate any of my prompt. It just gets me. It knows what I want and it just serves it up to me in a platter and off I go, you know, doing my thing. Um, now OpenRouter sits on top of like, kind of like the query layer, right? So you have all these people writing all these weird and wonderful prompts and kind of routing it through, um, on towards like different AI models. You hold all of that data, or maybe you have access to all of that data. And I know you have something called private chat as well, where you don't have access to it. Talk to me about like what OpenRouter and what you guys are thinking about doing with this data. Because presumably, or in my opinion, you guys have actually the best mode, arguably better than ChatGPT, because you have all these different types of prompts coming from all these different types of users. For all these different types of models. So theoretically, you could spin up some of the most personal AI models for each individual user if you wanted to. Do I have that correct? Or am I, you know, speaking crazy?

No, that's true. It's, it's, um, something we're, we're thinking about. Uh, we, by default, your prompts are not logged at all by, uh, or we don't have prompts or completions. For new users by default. Um, you have to toggle it on in settings. Um, but, uh, you know, like, the result— a lot of people do toggle it on, and, um, as a result, I think we have, like, by far the largest multimodal prompt dataset. Uh, but what we've done today, we've barely done anything with it. We classify a tiny, tiny, tiny subset of it, and that's what you see in the rankings page. Um, but, uh, what it could be done on like a per-account level is really, um, like 3 main things. One, memory right out of the box. You can, you can get this today by like combining OpenRouter with like a memory as a service. There's like a couple companies that do this, like Memzero and SuperMemory. Um, and we can partner with one of those companies or do something similar and just like provide a lot of distribution. And, and that basically gets you like a ChatGPT as a service where the model— where it feels like the model really knows you and, and context automatically gets, uh, the right context gets added to your prompt. Um, the other things that we can do are like help you select the right model more intelligently. Um, there's a lot of models where there's like a super clear like migration decision that needs to be made. And we can just see this very clearly in the data. But we, right now we just like, you know, we have like a channel or like some kind of communication channel open with the customer. We can just tell them like, hey, and we notice you're using this model a ton. It's been deprecated. This model is significantly better. You should move this kind of workload over to it. Or like this workload, you'll get way better pricing if you do this. Um, and, and that's basically like, that's the only sort of guidance and kind of like opinionated routing we've done so far. And it could be a lot more intelligent, a lot more out of the box, a lot more built into the product. Um, and then the, the last thing we can do, I mean, there's, there's probably tons of things we're not even thinking about. Um, but like getting really, really smart about how models and providers are responding to prompts and, uh, showing you just the really coolest data, just like telling you what kinds of, of prompts, um, are going to which models and how those models are replying and just like characterizing the reply in all kinds of interesting ways. Like, did the model refuse to answer? What's the refusal rate? Did the model, um, did the model like successfully make a tool call or did it decide to ignore all the tools that you passed in? That's a huge one. Um, did the model like pay attention to its context? Did, uh, you know, did, what did, did some kind of truncation happening happen before you sent it to the model? So there's all kinds of like edge cases that, that cause developers' apps to just get dumber and, um, and, and they're all detectable.

I'm so happy you said that because I, I, I, I have this kind of like hot take, but maybe not so hot take, which is I actually think all the frontier models right now are good enough to do the craziest stuff ever for each user, but we just haven't been able to unlock it because it just doesn't have the context. Sure, you can like attach it to a bunch of different tools and stuff, but if it doesn't know when to use the tool or how to process a certain prompt, or if the users themselves don't know how to like read, you know, what the output of the AI model themselves, like you just said, you know, you need, we need some kind of like analytics into all of this, then we're just kind of like walking around like headless chickens almost, right? Um, so I'm really happy that you said that. Um, one other thing that I wanted to get your take on, on the, on the data side of things is. I just think this whole concept or notion of AI agents is becoming such a big trend, Alex. And I've noticed a lot of frontier model labs, um, release new models that kind of spin up several instances of their AI model and they're tasked with a specific role, right? Okay, you're going to do the research, you're going to do the orchestrating, you're going to look online. Via a browser, blah, blah, blah, blah, blah. And then they coalesce together at the end of that little search and refine their answer and then present it to someone, right? You know, Grok4 does this, Claude does this, and a few other models. Um, I feel like with this data that you're describing, um, OpenRaja could be, uh, or could offer that as a feature, right? Which is essentially you can now have super intuitive context-rich agents that can do a lot more than just talk to you or answer your prompts, but they could probably do a bunch of other actions for you. Is that a fair take or is that something that maybe might be out of the realm of OpenRoutr?

Our strategy is to be the best inference layer for agents. And, um, what we, what I think, what, what I think developers want is control over how their agents work. And, uh, and our developers at least, like, want to use us for— as like a single pane of glass for doing inference. Um, but they want to, like, see and control the way an agent looks. An agent is, is basically just something that is doing inference in a loop and controlling the direction it goes. So, um, what, what we want to do is just like build incredible docs, really good primitives that make that easy to do so that, you know, like I, I think like a lot of our developers are just people building agents. And so what they want is they want the primitives to be solved so that they can just keep creating new versions and new ideas. Um, without worrying about, like, you know, reimplementing tool calling over and over again. And, um, and, and, and so, like, at least for— I— this is, like, a— it's, it's a tough problem given how many models— there's, like, a new model or provider every day, and, uh, and people actually want them and use them. So, uh, to standardize this, like, make, make these tools, like, really dependable, um, that's kind of, like, we want to focus. And, uh, so that like agent developers don't have to worry about it.

As we level up towards closer and closer to getting to AGI beyond, I'm curious what OpenRouter's kind of endgame is, if you have one. What is the, the master plan where you hope to end up? Because the assumption is, as, as these systems get more intelligent, as they're able to kind of make their own decisions and choose their own tool sets, what role does OpenRouter play in continuing to route that, that data through? Do you have a kind of master plan, a grand vision of where you see this all heading to.

You're saying like, as agents get better at choosing the tools that they use, what, what becomes our role when like the agents are, uh, really good at that?

Yes. Yes. And like, where do you see OpenRouter fitting into the picture and what would be the best case scenario for this, this future of OpenRouter?

Right now, OpenRouter is a bring your own tool. Platform. Um, we don't have like a marketplace of MCPs yet. Uh, and, and I, I do think like a lot of the, I think most of the most used tools will be ones that developers configure themselves. Agents just learn, like they're given access to it. Like, I think like a holy grail for, for OpenRouter is that the, the ecosystem is going to like— basically, my prediction for how the ecosystem is going to evolve is that, um, all the models are going to be adding state and other kinds of stickiness that just make you want to stick with them. So they're going to add server-side tool calls, they're going to add like, um, you know, web search that, that is stateful, or add memory. They're going to add all kinds of things that that try to prevent developers from leaving and, uh, and increase lock-in. And OpenRouter is doing the opposite. We, we want like developers to not feel vendor lock-in. We want them to feel like they have choice and they can like use the best intelligence even if they didn't before. You know, it's never too late to switch to a more intelligent model. That would be like, you know, a good always-on outcome for us. And so what I think we'll end up doing is like partnering with other companies or building the tools ourselves if we have to, so that developers don't feel stuck. That's how I, you know, there's a lot of ways the ecosystem could evolve, but that's how I would put it in a nutshell.

Okay.

Now there, there's another personal question that I was really curious about because I was also right there with you in the crypto cycle. Um, when NFTs got absolutely huge, was a big user of OpenSea, and it was kind of this trend that went up and then went down. And, uh, NFTs kind of fizzled out. It wasn't as hot anymore. And AI kind of took the wind from the sails and it's completely separate audience, but a similar thing where now it's the hottest thing in the world. And I'm curious how you see the trend continuing. Is this a cyclical thing? That has ups and downs, or is this a one-way trajectory of more tokens every day, more AI every day? Is, do you see it being a cyclical thing or is this a, a one-way trend towards up and to the right?

And NFTs kind of follow crypto in an indirect way. When crypto has ups and downs, NFTs generally lag a bit, but they, they have similar ups and downs. And, um, and crypto is an extremely long-term play on like building a new financial system. And there are so many reasons that that's not going to happen overnight. Um, and, and they're like, it's very, very entrenched reasons. Whereas AI, um, there are some overnight business transformations going on. And, uh, and the reason AI, I think, moves a lot— one of the reasons that AI moves a lot faster is it's just about making computers behave more like humans. So if you are— if a company already works with a bunch of humans, then there's, you know, there's some engineering that needs to be done. There's some, like, thinking about how to like scale this. But, but in general, I think that it's not— it, like, after seeing what can be possible, um, inference will be the fastest growing operating expense for all companies. It'll, it'll be like, oh, we can just hire high-performing employees at a click of a button, and they're— they all perform predictably, um, they're all AI, and, um, and we can measure and they work 24/7, they scale elastically. It's like, you know, it's not that hard. It's not like huge mental model shift. It's just like a huge upgrade to the way companies work today in most cases. So it's just completely different from crypto. There's, there's like, other than both being, you know, than NFTs, I mean, other than both being new, they're fundamentally very different changes.

You're probably one of very few people in the world right now that has crazy insights to every single AI model. Definitely more than the average user, right? Like I have like 3 or 4 subscriptions right now and I think I'm a hotshot, but you get access to like 400 and what is it? 57 models right now on OpenRouter. So an obvious question that I have for you is, I'm not going to say in the next couple of years because everything moves way too quickly in this sector, but over the next 6 months, is there anything really obvious to you, um, that should be focused on within the AI sector? Maybe it's like the way that certain models should be designed, or perhaps it's at the application layer that no one's talking about right now. Because going on to, like, going on from our earlier part of the conversation, you just pick these trends out really early, and I'm wondering if you see anything. And it doesn't have to be OpenRaptor related. It could just be AI related.

I've seen the models trending towards caring more about how resourceful they are than what knowledge they have in the bank. Not all of, I feel like a lot of the application, I think the model labs, maybe a lot of them, I don't know how many of them really deeply believe that. But, you know, a couple of them talk about it. And I don't think it's really hit the application space yet because people will, will ask ChatGPT things and if the knowledge is wrong, they think the model is stupid. And that's just kind of a bad way of evaluating a model. Like whatever knowledge a person has, whatever a person like recalls happened at a certain time, like, does not— it's not a proxy for how smart they are. Um, like, the, the intelligence and usefulness of a model is going to trend towards how good it is at using tools and, uh, and, and how good it is at, like, paying attention to its context of a long, long, long, long context. So it's like, it's, it's total memory capacity and accuracy. Um, so I think those two things need to be, like, emphasized more. Um, the, the, like, it might be that, that models pull all of their knowledge from, like, online databases, from, like, real-time, uh, scraped index, indices of the web, along with a ton of real-time updating data sources. And they're never— they're always kind of, like, relying on some some sort of database for knowledge, but relying on their reasoning process for tool calling. We put it in, we spend probably the plurality of our time every week on tool calling and figuring out how to make it work really well. Humans, the big difference between us and animals is that we're tool users and tool builders. And that's where human acceleration and innovation has happened. Um, so how do we get models, uh, creating tools and using tools very, very effectively? There's, there's very little— like, there's very, there are very few benchmarks. There's very little prior art. There's the TauBench, um, for measuring how good a model is at tool calling, but there's— and there's like maybe a few others. Um, there's SWE-Bench for measuring how good a model is at, at, um, multi-turn programming tasks. Um, it's very, very hard to run though. It costs like, you know, for, for Sonnet, it could cost like $10,000 to run it. Um, it's like the user experience for kind of like evaluating the real intelligence of these models is not good. And so like, I love— as much as we don't have benchmarks listed on OpenRouter today, I love benchmarks. And I, I think like the app ecosystem and like developer ecosystem should spend a lot more time making very cool and interesting ones. Also, we will give credit grants for, for all the best ones. So I highly encourage it.

Well, Alex, thank you for your time today. I think we're coming up on a close now. That was a fascinating conversation, man. And I think your entire journey from just non-AI stuff, so OpenSea all the way to OpenRouter, has just been a great indicator of like where these technologies are progressing and more importantly, where we're going to end up. I'm incredibly excited to see where OpenRouter goes beyond just prompt routing. Um, I think some of the stuff you spoke about on the data side of things is going to be fascinating and arguably one of your bigger features. So I'm excited for future releases. And as Josh said earlier, Um, if GPT-5 is releasing through your platform first, please give us some credits. We would love to use it. Um, but for the listeners of this show, um, as you know, we're trying to bring on the most interesting people, uh, to chat about AI and frontier tech. Uh, we hope you enjoyed this episode. And as always, please like, subscribe, and share it with any of your friends who would find this interesting. And we'll see you on the next one. Thanks, folks.