From OpenSea to OpenRouter: Alex Atallah’s Masterplan for LLMs | Data Driven NYC

I'm Alex Atallah. I'm the CEO and co-founder of OpenRouter, uh, a language model router and marketplace. Um, at the end of 2022, uh, I, I had, I was the co-founder and CTO of OpenSea, the first NFT marketplace, but I was winding my time down there and looking to start something new. And ChatGPT had just come out and my curiosity was piqued. So I had like one central question back then, which is, you know, will our language models going to be a winner-take-all market? At the time, OpenAI was just vastly far ahead compared to everybody else. There were basically no usable language models aside from the DaVinci 003 API and ChatGPT itself. There was Cohere Command, and then there were a lot of open source language models that were barely, barely usable, really not truly usable. And then Llama came out. It was a good start, but still not something that was substantially composable by developers, at least not to most developers. But then in March of 2023, a group at Stanford generated a bunch of what was called synthetic data, meaning they took ChatGPT and generated really good conversations, and they used that to take Meta's first open source model, Llama, and fine-tune a new creation called Alpaca. And when I saw this model, I was like, wow, this is different. Suddenly it, it only took $600 to make the whole model, to generate all the data and to fine-tune and create Alpaca. But this model was completely usable in the airplane. You, you could, you could just download it on your computer, turn the internet off, and learn about the world and ask it knowledge questions and even generate code, really basic but decently functional code. And so this was kind of like a turning point, um, in the whole language model community. And at this point I was like, okay, it, it makes sense to at least collect all of these things in one place so that people can explore them. Because if this is as easy as it was to make, there's gonna be hundreds, thousands, maybe millions of language models that grow in the future. And I have no idea how to distinguish one from the other or what makes any of them better than using the, the market leader. So that was how OpenRouter came to be. And OpenRouter started as a, a, a router and marketplace for language models where you can just Explore them, figure out what their prices are, how much data they can consume, which is called the context length. You know, can they take in images or is it just text? That's the modality. Um, how much does the prompt cost? How much does the completion cost? Uh, what architecture is it based on? And then down on the very bottom of the screen here, what categories of prompts is this thing good at? Because all of these models are kind of like black boxes. Even the open source ones, you get 7 billion integer or, you know, 7 billion floating point numbers and you, you, you don't really know what makes the model better than all the others. So what we did is instead of letting you, you know, sift through benchmarks to figure out what tests a model has passed and, and then these tests are kind of arbitrary and they expire. And they're not very durable. We wanted to show people real market data, like what are people using these models for, which categories among power users are really popular, so that you could learn from an enthusiast community that we build ourselves. And so we started cultivating a couple different communities of enthusiasts testing out these new models and seeing what they would be good at. And it grew fairly stably at a very good grip, about 10 to 100% growth month over month for the last 2 years. We, you know, a couple weeks ago crossed 1 trillion tokens in process per week, which is, a token is about a word. And we show all this data publicly. You might ask like, who is actually using us and, and why. At the beginning, we saw a lot of apps focus on storytelling, and that was because OpenAI really struggled at doing some kinds of stories in the early days. There was a lot of moderation that had a lot of false positives. For example, you couldn't like generate a story, a detective novel where somebody is murdered in the middle of the, of the chapter. because the model would refuse to generate the scene. And so initially we saw some traction around roleplay and storytelling for people trying to get a, like, like get a model that would be able to generate those parts of the story. Um, later we saw a shift towards coding, and this really picked up over the last 7 months. And so now you can see the top couple of apps, Klein and RuCode, those are both similar to Cursor, if you've heard of it. Um, and, uh, you know, the first breakout AI consumer app was Character AI, which was a, you know, a roleplay experience. The second major consumer breakout app in AI was Cursor, a code generator. And, and we see like that reflected very clearly in the apps that rank on OpenRouter as well. So how does it work? We built one API with zero, near zero switching costs. That lets you just change a URL if you're, if you already have code that works with OpenAI, and then change a single line of code to explore new models, that line 12 there. And we provide lots of developer quality of life features right out of the box. Prompt caching, structured outputs, tool calling, message transforms, context compression, automatic context compression, um, web search. Uh, we ensure completion. So if the provider for some reason doesn't work, you don't get charged. Um, provisioned API keys and a lot more. And so we do all of this and we try to standardize the experience to give you the best in class, even if you're using a niche model or something unusual. Um, we, for, for many models, uh, there's a lot of different prices. For all the open source ones, you really don't want to feel like a chump getting the worst price on the market when you can get the exact same thing somewhere else. Language models are often a commodity in that sense. Um, and then there's some models where it's the same price everywhere, like Claude 3.7 Sonnet. And, um, in those cases, we try to get you the best performance. Which provider is actually nearest to you in the world, which one is giving you the fastest tokens per second. And part of the benefit of that too is that we can boost your uptime. So we can always give you a better uptime than going directly because you can use your own key, add your rate limits to ours, and in the worst case, if everybody is, if the original provider is down, we have a list of fallbacks that we try. That gives you the, that give you the same experience. And then you can see our chart here, which we put on all of our model pages that show the uptime boost you get with OpenRouter, which changes depending on the model, the market, the supply, and the demand. We also provide a chat room, which is like a developer playground, allows you to really easily compare a ton of different models top to bottom against your prompts. And copy and paste the code to get started right away. And then we give you observability and accounting, a standard experience that works across all models. So you can see your activity, what usage you're doing, broken down by model, and you can click on individual rows in your activity and get an API so that you can query to see how much it costs, how much you got, how much caching happened, if you got any discounts, you can opt into different kinds of discounts through OpenRouter, what the data policy was of the underlying provider, you know, why it stopped exactly, all kinds of things like that. We have organization controls. If you're part of a larger company or if you're bringing OpenRouter to work, we try to make that a really seamless experience. So you can set up an organization, set one list of policies, like providers you want to ignore, whether you want to allow your prompts to go to any providers that may train on them, whether you want your prompts to ever be logged by us, and then a lot of other features. These are all default off, but you can always opt in if you want. And that's it. it's a, um, start to what I think could be one of the, the biggest types of APIs in the world. And we have S3 for storage, but soon intelligence will be just something everything needs to get sourced. And we're trying to be the best place to source intelligence where you can plug in, um, your credentials and immediately get better prices, better performance, better uptime, and best model selection. So you're going to build the best products without having to worry about sourcing the right intelligence.