Alex Atallah of OpenRouter argues most people are barely scratching the surface of what LLMs can do, especially for domain-specific work like product management and calendar scheduling. OpenRouter's AutoRouter picks the best model for each task based on real performance metrics, not sponsorships. Smaller, cheaper models often work fine where people default to the latest big ones.
Chapters
I think it's absolutely insane how underutilized LLMs still are. There are, I mean, on top of just running into a lot of repetitive tasks that even engineers continue to do today, like the, I think the most exciting sort of glimpse into how many more tasks can be automated is the new types of training data that the model labs want. And they're really interested in figuring out how to be a product manager. What kinds of things do you need to know? How do you take a bunch of different contexts and distill it into action items? Or better ways of running agent workloads, understanding somebody's calendar very deeply. And these, these tasks, like after a little bit of time, they start to show up in the models. And what our job is, but what we want to do is make that work visible to everyone. You know, if a model does a really good job managing your calendar, that should be clear in the rankings using empirical data based on how people are using it. And, uh, and then, and also other signals that we, we kind of understand over time. So, um, but we're like, you know, there's, there aren't that many categories that like Model Labs are known to optimize for today, and it's just going to blow up.
So I think the, I mean, I think it's, it's pretty amazing how, how much more work there, there continues to Just getting that done though, for folks who maybe are a little bit less ahead in this world as we are, you're saying that LLMs are going to get better at domain-specific tasks by better training by their makers. And so essentially we're going to see LLMs perform better at different categories of knowledge work, product management and so forth, and not just drafting legal documents and writing computer code.
Yeah, I mean, I, you know, I'm not training models myself, and so I'm not the best person to ask about the exact ways they're going to improve. This is kind of my speculation and hope. But I think the, like, that's like a big signal that there's room to grow. The fact that nobody is tracking improvement on those things yet. You know, a key kind of muscle to build when you're entering the AI space is benchmarking. Everything is about evals and benchmarks. And if you make claims without having an eval or a benchmark, what are you really saying? And there's a lot of like, you know, there's just a lot of value to unlock by creating a benchmark. And it also guides the models in that direction when you do.
So evals and benchmarks are very useful. There's other vectors you might use to choose which model, which provider at which time. You guys built something called AutoRouter that essentially does some of that work for your customers. I presume today it's an unbiased setup in which you guys are only selecting for the best option for your customer, but is there a way in the future in which people could sponsor that to get more people to try out their models for certain use cases?
It's a good question. So far, we've stayed away from letting any providers sponsor more traffic. Um, our goal is to be like a strong, neutral Switzerland-style gateway that gives everybody a fair shot based on, on, you know, real-world metrics and, uh, and, and real-world benchmarks. So, but I do think that there will be interesting ways to like shift people to new models in the future. We, for example, like, we do a little bit of model recommendation today, but very little. You'll start to see that ramp up in 2016. You know, if you're— 2026. 2026.
I know we're getting old, but like, we have to at least get the decade right. You're good. Keep going.
If you— yeah, yeah. There are all kinds of workloads that are just like flooding models that are way too powerful and don't, you know, where you could save a ton of money.
Yeah, you don't need to use GPT-5 High if you can use, you know, 4.1 or whatever.
Yeah, yeah, yeah. Exactly. And having like a really wide range of models and a lot of data about how they get used, I think helps there. But, uh, but there's also, I think, good ways of, of like making very interesting model routers that we'll experiment with.
Founders, let's be real, finding the right developers is really hard when you're trying to run and scale your startup. But Lemon.io can save you time, money, and a ton of headaches. They've pre-vetted high-quality developers, and they've ensured that they're experienced, result-oriented, and that they're seeking competitive rates. These are hand-picked devs with a minimum of 3 years of on-the-job experience, and only 1% of applications to Lemon.io are accepted into their program. So forget the long drawn-out hiring process. Lemon's gonna match you with a skilled developer in just a few days. And they're not just recruiting talent either. They're helping you integrate them into your team. And if anything goes wrong, Lemon will find you a replacement developer ASAP. We've had a ton of Launch Founders work with Lemon.io. So go to Lemon.io/twist and find your perfect developer or tech team in 48 hours or less. TWIST listeners get 15% off their first 4 weeks. That's lemon.io/twist. L-E-M-O-N.io/twist.
