How To Use Open Models Effectively

Open models are on everybody's mind right now, especially with them being so good. GPT-3 just came out and we see a commitment from a lot of people to like keep making these open models, like NVIDIA. People are concerned about costs and then there's ownership. People in certain cases, they want data privacy. And the question that comes up is, okay, when should you use open models? How do you make that cost trade-off? What are the hidden costs? What are the benefits? How should you think about it? How do you serve these models? And so forth. And Zach has been living in this world for like more than a decade.

Thank you, Hamel Husain, for the very, very warm introduction. So my name is Zach Mueller. I am now head of developer relations at Lambda. And essentially this is gonna be a presentation on how to eyeball open models, where they fit into your pipelines. So we're gonna go deep into like, how do you think about these decisions and a little bit into some ideas on how that can affect cost. Cost compared to a year ago, especially on the smaller model size, we have like this vast improvement in intelligence per parameter, right? So Codex, Claude, all these closed source harnesses, they're fantastic. They are day to day. But in general, most people probably could get by with open source models for 90% of their tasks. Some of it's economical, right? You can maybe run it on like a small GPU at home. And so it all boils down to sort of what is this decision scope? When do you use open models? When do you don't? So first, let's talk about where it's going to run and how it's going to work. So right now there's a lot of chatter and discussion about OpenClaw, Hermes Agent. We'll call them user interface-oriented harnesses. So what that means is you're not sitting in front of a chat window, you're not sitting there talking to it on your phone per se, right? And then you have the other side of it, which is your coding harnesses. So Codex, Claude, Pi, where you have these frontier models living inside of their subsidized token harnesses for you to do day-to-day work with. Both of them, both of these can be run by open models to various degrees of success, but obviously some are built more towards allowing open models to be successful than not. So in that same vein, right, smaller models are generally good enough now for a lot of things. Big models are slower. OpenAI is having to bring on Cerberus in order to make, uh, GPT-5, GPT-6 much faster because the model is gigantic. We don't even know how big it is. Rumors are like, you know, tens of trillions. We don't really know for certain. On top of that, a small model nowadays is not just how you would think a year ago where it's, oh, if I'm sacrificing model size, then it's probably gonna be some 20 billion parameter model that's hyper-optimized for my task to do this one thing really, really well. And that's not the case anymore. We have small models that are very, very good at generalizing problems and working through those problems that, you know, we used to reserve for the big models in the closed labs. Now, it doesn't mean that every small model is built equally, nor does that mean that you can perform magic out of a small model. So when I think about the open source models that are impactful in my day-to-day work, this is generally going to be Quen non-MOE, so that's gonna be 27 billion parameters. Uh, the Quen MOE is at like 30 billion. Anything bigger than that in the Quen models, lose is a harsh word for that, but there are better candidates, we'll say. GLM 5.2, this is almost my daily driver now. Got it deployed upstairs even, and I just use it constantly. Another one not on here that I do wanna bring up is Deepseek V4 Flash. Big Deepseek. Well, we'll say didn't have the largest welcome whenever it came out because it was this big model that wasn't state of the art, that Deepseek train that we waited, you know, a year for since Deepseek V3, and it didn't quite have the Deepseek moment. And instead what we've found is the smaller variant, it's still big, but that in and of itself is a very capable model as like a Haiku replacement. And then we have like KIMI, your Minimax, your Nimotron, GLM size, but also sort of in its own category, I would argue. Like Minimax is questionably, you know, more commercial now. KIMI, I'm a big fan of KIMI for writing and a view for other tasks. So I'm very biased to that, but also KIMI, $3 trillion now. And then NemoTron, right? Built in the US, built on US data, largest US open source model at the time. You know, there's different benefits to shop around relative to performance that you need for a task, your ability to run it in general, as well as the size and scope of what you really need it for. So we're gonna do a little bit of math just because I want you to have a rough idea on things. So if we think of Qwen non-MOE, that's about 27 billion parameters. And so if we think about what that takes to run it at home, typically you would run it in, in 8-bit or 4-bit. And so if we go down to 4-bit, which we'll get into quantization in a moment, you'll take 27 times 1, so that's gonna be 27, divide that by 2, and it takes about 14 gigs. And so you can run Qwen 27 billion, which has by far, it's Qwen 3.6 27 billion, the most capable model for an everyday agent. Hundreds of people I know are running Hermes Agent purely off of this one model. And I was running Hermes Agent off of this one model for a few months. And so that gets us to about 14, 13, or 12-ish. And then you add a little bit of overhead for activations. But it essentially can fit on a 16GB card. Uh, and then if you go into like 8-bit, now you're in like a 5090 territory. But essentially you can sit there with some of the cheapest hardware, run a small model, and it's pretty capable of, you know, doing website lookups, talking to you, finding documents, calling different APIs, not per se coding, and like, you know, generating these things that you would want to to replace Codex with, but for the small things at home that you want to do, like checking emails, it's, it's a very capable model. Qwen did a phenomenal job, which is Alibaba, in training these small models in this way. Now, if we get to like GLM, GLM is like 800 billion parameters. I am running it on 6 96GB cards. That rig alone is— cards alone are like $60,000. For the average guy at home, that's not really feasible, but you know, a Big Mac could run it. And that is like a daily driver workhorse for coding. It is very, very well optimized and trustworthy for code problems. We'll talk about more in a moment on why. And it's a very strong workhorse for running, like just running it through your job. What my typical setup is, is I'll have like 2 to 3 models running locally. So that's, you know, Qwen 27 billion. I'll also do Liquids models. They're very small mixture of expert models. That handle the MCP inside of Claude Code. And this lets me save Claude Code tokens because the Liquid model does it enough and stays on my computer. When you invoke like /mcp and you call like the Notion MCP, what happens is you send a prompt in that Claude has to go and translate to MCP to a function call to go do something. And the thing is, when you're in Claude Code, it loads all your MCPs in. Always, and so it is always in context. And so you can essentially change how it's being used by having the entire backend of Claude for that specific problem be Liquid instead. And what I found is not only is it cheaper, cuz you know, it's running for free on my computer, was also able to reduce the time it took to do the whole MCP by I think like a tenth of the time due to being able to optimize it that you can't do.

You have like an MCP that's kind of just this layer that is, uh, Kemi driven, decides what MCP to call and you replace that, you replace all your individual MCPs with this one, or how does it work in practice?

You essentially have a wrapper MCP for the Notion MCP. And then I create a new Claude Code instance with it as the backend. And so once Claude Code is being ran by this 8 billion parameter model, I can just use it to do Notion calls and it works very reliably. And like, you know, some other places you don't want it to be and that's okay. I'm finding a balance with it.

You can switch the model in Claude Code. Like you can point it at any API you want. They have like a environment variable. Variable you can set, I think. So are you saying like, do you like for the whole session of Claude code, you switch a model to something else, to this like wrapper thing, and then that routes to whatever model you want?

Or— Essentially another way to think about it is like I have Claude called Claude-C and that Claude-C is from a very specific folder, which invokes it with already injected environment variables to have the liquid model set up to go do the thing. But what I'm saying though is like these open models can take where most of your tokens go. And redirect it into something that you host yourself or on the cloud in your day-to-day workflow already and save money because now your cloud bills are lower, especially if you're like on enterprise or the big plans where it's not subsidized. And also, you know, not have to worry with security about like, well, hey, I heard I can, I can, you know, save money by using like Liquid or GLM. Can I go point our company data at OpenRouter because it's cheap and it works? And no, of course not. Cause there's security risks. And that, but if you can find a way to run it yourself, okay, now it's a little more nice conversation because it's—

and just to be clear, like the OpenRouter has security risks because OpenRouter just routes to different providers and those providers can be hosted anywhere, including outside the country. And then those routers can do something like, okay, you know, have instructions to like upload all your code somewhere. But even in the American, like Groq did that, right? Like there's American versions of that too. No shade on anything Chinese. You just have to be careful.

It's like when you're thinking about it as a company, right? You want the lowest risk security variable and a third-party API where you don't have an enterprise deal with and you don't have a known secure, like, way of accessing it, then the safest option is to find a way to run it yourself. And if you can find a small model that can do the specific thing that you want to do very well, which there's a lot of them now, it's very easy to just have that replacement in. So getting back to this, so we have GLM, KIMI, Minimax, and Nimotron, right? So in the whole scope of things, GLM I target for coding because it's a very strong coding model in general. It's very efficient. It doesn't get distracted. It doesn't make small mistakes. KIMI I don't use as much because it's even too big for me to run things a lot. Cause it's like, it takes an 8x B200 to do it. That doesn't mean it's bad. It's a fantastic model. I love running it. Just an 8x B200 is, you know, terabyte. And a half of VRAM, which is not the most accessible thing in the world. Minimax, I don't do as much because they've had the more enterprise license change. They're still fantastic models, but since 2.6, I haven't kept an eye on them too much. And then NemoTron is great because it's passed through American systems. So it is an American trusted model and it's much easier to get past enterprise. Say like, hey, scary Chinese model, which aren't really that scary. Here's an American one instead that gets it kind of close that we can reliably do.

What coding harnesses are you using? GLM, Nimotron? Are you using Claude code?

I've actually stopped using Claude for 98% of everything now. I am now Codex pilled.

Definitely. Yeah, it's really good.

OpenAI's models have just by and large gotten a lot better. And also for my own day to day, I fight Fable more than doing work. I don't fight Codex and Sol and everyone because I'm not triggering any security flags. We'll go into that in a moment of how I came to You mentioned you use GLM a lot.

Okay. What coding harness are you using?

Usually Py, but specifically now I'm switching to NumenaCode or NCode. The other thing that you need to worry about when you're deploying models locally, cloud adjacent, is what your constraint is, because that can be latency, which is time to first token. That can be tokens per second, where some things don't matter how fast it is. And then of course cost. So if we look at a visual, 20 tokens per second is like what I would call baseline usable for small tasks that happen in the background right I don't have to keep an eye on it, right? Because it's going to take a while. It is not fast at all, but it's good enough to go do something and trust that it'll run. So I can do like CPU offloading of a gigantic model, hit 10 to 20 tokens per second, and like make sure I have the intelligence I need without spending $200 grand on GPUs and getting decent speed that can do tasks well. 50 tokens per second is more what I try and hit as a usable workflow. Because it's about Opus level. Like if you've used Opus as your main coding agent for a while, it's about this fast. Sonnet and Haiku will be faster. Smaller models will be faster. But if you can get the model you want at like 50 tokens per second, it is very, very usable at home. I think in one weekend I was using GLM 5.2 to optimize its own deployment from 20 tokens per second to like 47 now. And that took a day because it was sitting there optimizing itself, but I've been running it. I believe I've processed. 2 to 300 million tokens already in the first, like, week and a half through it. So it's very, very usable. Then you have 100 tokens per second, which this is ideal because, you know, it's fast enough that you can't read it now almost. This is great because usually you want this for subagents. So, like, you want your subagents to run really fast, make sure that they can process the code fast enough and write the code fast enough. And then for fun, a while ago, Sam Altman mentioned that Cerebras and OpenAI were working together to make sure that GPT-5.6 can run at 750 tokens per second. Here's what that looks like. When we hit this point, it will be unfeasible to even check what the model is writing because this level of speed is like Gastown levels. It's insane. And we're certainly not going to get that at home. And I'm not even sure if I could make use of 750 tokens per second because of how much you're going to need to like update and talk to the model. When I tweeted about this, there was a very good comment where it's like, imagine a world where 750 tokens per second is like the chain of thought. So as the model is thinking and doing a bunch of nonsense, and then it's presented to the user at 50 tokens per second. That's more doable because it reduces the time for turns in like a chat time in a chat window. But your screen is not looking like this 24/7 and then stops suddenly. That's a small little tangent, but like basically find your target on how fast of a responsiveness you want, which dials back to either a hardware condition or optimizing on the software side. So with kernels., and so forth, which we'll talk about kernels at the very end. But the next question is in relation to cost quantization, right? Because ideally we run these models in the native precision that they get released in, but also that tends to be the most expensive, right? So a, for instance, KIMI-K2 is a 1 trillion parameter model in FP8. It requires, uh, 8 or 16 B200s to run. So that's, you know, 1 to 2 terabytes of VRAM. Ideally you want it, especially for when you're doing like reinforcement learning and you're doing rollouts, you want to have the precision of the model you're doing RL against to match the precision of the model that you're training. There's a thinky paper about this, uh, that went viral. So in general, great, but also is the most expensive solution because we can directly map like how stark of a performance drop you get by not fixing the this problem. The next level that I tend to consider is NVF-B4 and MXF-B4. MXF-B4 is for Hopper and when you don't have like Blackwell-level hardware. Generally, FP4 is near lossless. A lot of models release in FP4 now. I tend to like NVF-B4 because I have Blackwell at home and I have Blackwell with work, so use the quantization that fits the best for it. Usually when I do the quantizations, NVIDIA will release their own NVFP4 quantized weights. And usually it's through quantizable training and generally they try and keep it pretty lossless. Also because it's optimized for NVIDIA hardware, it'll work well. And so right now my general setup is like I'll run GLM 5.2 NVFP4 before I deployed it upstairs on like half of a B200 cluster and then KIMI-K 2.7 code on the other half of like 8 B200 GPUs, uh, because each of them will fit on 4 now instead of needing 8 in FP8.

You mentioned a data center you have upstairs. Is it worth it for most people? I mean, I understand like you are a tinkerer. Your job is to get really deep into inference and experiment with them, but are you gonna get into that maybe? Like where should people be hosting these things? How they should be hosting these things?

Yeah, so for transparency, I have 6,000 Blackwells upstairs of various models. It essentially boils down to, is it feasible that rather than buying a very expensive car, you instead increase your electric bill, potentially 2x-ing it? That is one way to think about if it's a good idea. The other way to think about if it's a good idea is, does the skills that you learn by self-hosting models help you further in your career. You know, Hamel mentioned I got started at Hugging Face, so I was doing Accelerate, which is multi-GPU training and inference. So from the beginning, like, I've had a reason related to my job to have a bunch of GPUs at home. This started out with like two 4090s and then escalated. I'm not saying it to the level of like Ahmad because His electric bill is ungodly. I'm pretty sure he's at like 39 4090s at home and like 6,000 Pros. It's disgusting. I love that for him. He can run so many models, so many things. But also it comes to an economical decision of, if you think about it from an economical perspective of, well, if I run the GPUs at 100%, it'll pay itself off after XYZ time. That is a flawed piece of logic. At this point, it is training and sharpening my skills. I don't even try and justify it anymore. Anymore, because in reality you run it maybe 40 to 60% of the time, right? Because it's going to be the time when you're working, maybe a little after work because you're tinkering, or maybe not even that because you stand up Hermes and you call it once a day, right? And so for most people that want to do this and want to try to do this, Spot instances are like a 2 to 3 month term commitment on like a GPU cloud where you know you want to serve like GLM, try it out just for yourself, make it run fast. Then that is a viable option because you're putting in like quarter or less of the initial cost of like doing something like this crazy rig at home on a computer that you don't own. So you don't have to do maintenance on it. Uh, and B, their electric bill, it's not your electric bill. And so really sit down and examine if you care about owning the model deployment for data reasons, where it's going and whatnot. Look at what you can run at home first. Then examine the various clouds, right? I'll be biased to Lambda because Lambda's— and look at the cost of what would take to run it for 2 to 3 weeks and track your token usage and see if you're actually using it like you hoped you would. And if you are, okay, either do more long-term commitments or go out and buy hardware because you're actually using it successfully. Otherwise it's a toy and it's a very expensive toy where you don't have a measurable impact of how the toy is changing your day-to-day life. I happen to work in a way where I can measure this and it is actually impactful. Hamel Husain is slightly aware of this, but I am building my own EA that tracks all of my inputs and predicts where I am forgetting certain things or how I'm gonna act in certain ways and make sure that it's always in my attention. And it's ran by like half a dozen open source models, a variety of sizes on a DGX Spark plus GLM plus a little bit of closed cloud. That's an extreme measure, but it's what I came to. But I'm also crazy. So take my own words with a grain of salt with that. So like it comes to a point where as long as you can find a way to measure, are you burning money or is it actually making a change? Then you can decide, is it worth, you know, finding a long-term commitment, finding the GPUs to own, or is Claude and Codex good enough for me and I don't care about the API cost.

Yeah. And there's also the other extreme, just use hosting provider that hosts the open models.

Exactly. So now it boils down to VLLM, SG Lang, and like Llama CPP. In general, I sort of ignore Llama CPP because a lot of the stuff I do on my rig is like batched. It's multi-user, be it either my friends or my own subagents. So if we think about VLLM and SG Lang specifically, cause that's the territory you get into, they're more or less the same. SG Lang. Has some more advanced serving architectures. That's going to be like disaggregated inference, cache-aware routing, things that are useful deploying at scale to like 100 users or 1,000 users. While VLLM is more like, it has broader model support, quantization support, special kernels for specific hardware, and generally more stable hardware coverage for models.

And when I was serving models about 2 years ago, because I had been away from OpenModels for a bit, I remember VLLM was very easy to use. It was like, oftentimes one command where you put the model name and maybe some optional parameters and did a pretty good job making everything work. Is this still the case?

They've put a lot of work into making sure that out of the box it works really well. And then you can like go look at their cookbook and, you know, there's some recipes there. I also try and find some optimized recipes for like the cloud hardware. But generally it's like an out-of-the-box experience with VLLM is something they've really strived for. They call it the Oracle that picks and chooses the right kernels based on your hardware and the size of the model automatically. Perfect. But it's pretty good at picking the right kernel. SG Lang sometimes is better at picking the right kernel because the right kernels exist on each one. It's just sort of a fight of, well, it's the first 3 days a model came out. Yeah, they might not be the most stable at finding it out because we're finding out which kernels are actually best for the architecture as it goes live, but it's gotten better, I would say, over the last few months of out-of-the-box performance. So, you know, if it's at home, VLLM, if it's on like a single node, VLLM, the second you go to like multi-node where you're worrying about crazy serving techniques like disaggregated inference, which if so, why are you at this talk? You already know all of this, you know, then you want to scale to InfiniBand level clusters. And I mean, it's right now at home, I am having to do weird VLLM nonsense because I have different topologies because 4 GPUs live on one computer and 2 GPUs live on the other. I find fun challenges for myself because I choose not to do things out of the box. This gets us down to the end. How does this scale? Right. So we've talked about, you know, yeah, replace some harnesses, do some nonsense, have some custom things. How does it actually end up working though? Like, can I give you a tangible answer to this will work? XJDR, he works at Numena. Numena is a, a Neo lab that does frontier research. He built it himself. It is just himself and his agents off of his harness that he built. And I do not believe I know anyone that has used more tokens on open source models to do frontier research than him. So like my poster child of if he says something, I immediately go investigate it. And essentially what he's come to is that open weight models are good enough for about 90% of queries from 90% of people. People. They are just that good at this point. And I'm even considering like 3 trillion parameter KIMI here. Most of you, because I'm not in this bubble, aren't in that 10%. That 10% is like people at OpenAI, at Google, at Anthropic that are like pushing the frontier and trying to like crack these crazy math problems. Pretty much all of us, I can safely say, could get by with a setup that is purely open model. But that being said, XJDR's harness that he does right now, which is NumenaCode. This is what I've been using for the last few days and it's worked very well. It started as a fork of Claude code, but what he runs is essentially, so imagine the Claude model ideology, right? So you have Opus, which is very strong as a driver. You have Sonnet, which helps organize tasks. And then you have Haiku, which is a very fast agent-based model that can go like spread itself out and, you know, call multiple subagents at once. And essentially. He took Claude Code, ripped out all of his frustrations with it, and by the end he had an entirely different harness than Claude Code itself, which is why he could open source it. That's ran off of this model stack where you have SOL that is the driver, you have GLM 5.2 that is the implementer and like sort of the overseer, and then DeepSeqV4 Flash, which is writing all the code. And his loop essentially has allowed him to trust the code that comes out of this. Because the models are there and the models are good enough. Many, many months ago, xJDR, when Kamek 2.6 came out, he fine-tuned K6 for his own tasks, like his own harness. Transparency, he has an NVL 70B200 that he does this with. So like, I am very thankful that he is spending all of this money for doing frontier research. And essentially he fine-tuned K2.6 for his harness, ran it for a while, and for many, many months he's had this loop of he would try an open source model, let it drive for a bit, and then swap to a closed source model and then see how much code the new closed source model, the new frontier model removed. Right. And this is a good metric to understand how good open source models are because it's code maintenance, right? It's if the model rips out everything, then the ideologies of how they believe the problem should be attacked are entirely different. The same code from that K2.6 fine-tune exists today, even though he's driving SOL and he's driven Fable and he's driven all of these new models that replace that top model. And so that is why he can comfortably say that for 90% of people, open source is good enough. When it comes down to his own problem set, which is again frontier research, it's just barely above, like the closed source models are just barely above the capabilities of the open source models that it's not useful for him. That is also the equivalent of saying Steve Jobs can barely not run Microsoft off of open source models, so he's doing OpenAI. Like, I'm Johnny Bravo, you know, I'm not a CEO of Microsoft. I could probably get by with open source for everything I do. And now the conversation turns into like, well, find a hosting provider that can be secure for you or host it yourself and run a setup like this with an open source harness. That is performative for the task you want, because also these models are only good as the harness it's in, because they're one in the same. Article that talks all about Numena and encode. But basically what I want you to come away with is genuinely give open source models a shot at replacing some of your pipelines, uh, you know, based off of what resources you can do, right? If you need to do a quantization version of like a big model and run it on like a single H100, because that's all you can afford right now, then go try that., but genuinely try replacing these models and not just inject it in the backend of Claude or inject it in the backend of Codex and hope that it works. Like, give it the best chance for success. Py is an open source harness that is completely minimal that the models have like the biggest opportunity to do stuff with. I replaced 3 or 4 workflows from Claude code by using GLM 5.2 and Py.

So one question I have is there's so many model choices. And then like, you know, there's thinking levels and you have the closed models. And what I've been seeing the rise of routers, like that smart route for you. So like RAMP came out with one. There's some coding harnesses that like bake it in, like Droid and AMP. They bake in some like level of routing, like depending on your task. I think Cursor just came out with some amount of routing, like where they like based on the task to be done, it like routes different models. Have you used anything like that? Have you seen anything like that? Have you?

So one of the things with Numena is their harness is like Claude where it'll dictate the subagents to those models. So like if the task is hard enough, you know, it'll use Sol, otherwise it'll use GLM, otherwise it'll route things to like Flash. But the important thing there for you, the consumer, is KVCache and shared contexts. Because if you don't hit a warm cache, it gets very expensive very quickly. Because they're having to rebuild that cache from scratch every single time you switch context to a different provider. And so we'll never use a model router unless it has very explicit guides on, you know, I pick the models that it's routing to, the tasks that it is being worked on based on a recommendation, and then staying at one deployment provider for it. Because then it's useful because like the workload that comes from a coding environment is you know, if we think about like Prefill versus Decode, it's like a 20,000 character input because it's getting all of the stuff from your code base and like a small output. You want that to hit a Prefill cache. And if you have to reallocate that more than once, it's disgustingly slow. And so like, it makes sense if it's like chat models go to one place and coding models go to another place. But it needs to have like, I'm a control freak with that. So I don't think I'll ever use Magic Routers because I don't trust anyone to make a better— the agent swarm that I plan on giving that thing. So you'll see some thoughts on X probably in the next week, uh, of my thoughts with it. But my initial impressions is it's, you know, it's a Claude code fork. It's very simplistic to use. I used Claude for a year and a half almost, or whenever it came out. And so it felt like very at home, but it seems to be one of those harnesses where it's like, yeah, it is designed for a task like frontier research, but also that's not bad because one harness shouldn't be able to do everything. You want an optimized harness for specific problems because then you're giving the model better tools and how to do the things you want it to do.

What does it mean for a harness to be optimized for frontier research? I'm trying to picture that, like, okay, the harness is good at coding. Wouldn't it be good for frontier research? Like, what is it about frontier research you think that—

For frontier research, in my opinion, deep research matters more than anything. Because you need to research the task you're trying to accomplish, find the papers that relate to it, find the implementations that relate to it, and then start running experiments, reimplementing those papers until you find novel ideas from it. And so you want a harness that gives the model this search space in an efficient manner. That can be done through MCPs, of course, like Flywheel is very good for this because it generates like a graph for you of how the model is exploring a search space. But the more you can like let the model not fight all of these internal processes of like how the harnesses want you to do certain things and like have certain things in context at all time. And instead just to have the mission be do the one thing really well, it'll outperform you know, a general harness because it is designed for the task. It's just a matter of do you have the time and energy to go and either build that yourself through Py or, you know, try and identify which ones are good enough for you. Difference between Py and OpenCode? Uh, Py is absolutely minimal as a coding harness as a whole. OpenCode I think of as more like a big, bigger product sort of thing. How do you evaluate OpenLLMs? Vibes? Honestly, like I'll sit there for a little while, chat with it, read, uh, the people whose opinions matter to me, like xJDR. And if there's good signs that they're doing well, I'll go try it out.

Copilot has auto mode based on some research. Then there's a paper and, and, and I believe, or last time I checked, Pamela works on Copilot. Kimi has been the best in her experiments. Switch over Copilot.

That's very reasonable. Yeah, Copilot is. In my circles, we cry if a friend tells us about Copilot and we bring them to the light side of— I will happily do a plug real quick. I publish the LLM Index whenever a model comes out, which is step-by-step directions on how to deploy it on cloud GPUs, but you can take what is done here and extrapolate it. I also benchmark it in like tokens per second.

Anytime a model comes out, you have a page on this Lambda site. That tells you exactly how to deploy it.

To be more specific, it is models that I particularly am a fan of.

Pretty useful. So like, okay, so you find it, where do you go for this?

It is lambda.ai/entrance-models. There's one coming out for, there was an American lab that just released a 100 billion parameter model that could fit on a card. I'm blanking on their name. Florian would be very upset with me right now, but there'll be more model cards to come, especially with Quinn and Kimi. I'm working on getting the compute internally to Benchmark KIMI, cuz that's—

these recipes that you have on the site, like, is it resi— designed to run on a VPS or is it designed to work on Lambda? You just copy and paste.

It's Lambda. They're good general directions, don't get me wrong, but like I benchmark it on Lambda. There's a reason why I went to Lambda. Like it's, I use their GPUs, I think in my very first hackathon, if I'm not mistaken. And so you know what you get and you know the price you pay for it, but you know what you're getting and it's really good. Like it just always works. It's very reliable.

Are they all like Docker recipes?

Uh, usually I try and do Docker because there will usually be Docker image releases for the models. And also Docker is just what I stay in. I've debated about if I've wanted to do like UV instead, but run all of my servers through Docker at this point, if I'm running models. So I just do that. But also I am a man that runs like 7 or 8 different models at a given time and like swaps them out regularly. So Docker is very easy for me to like just pick one, swap it out and go from there. Can we reserve dedicated GPUs for open source model hosting on Lambda? I mean, you essentially are renting our GPUs. GPUs. So like you pay the GPUs, you don't pay like a discount because you're renting a particular model. You know, the aim is like you're renting the hardware. That's why I was like budget accordingly for renting hardware to test this out for a few months and a few weeks. But you know, you snag a node and then you spawn it up and Lambda has like, if you don't need 8 GPUs, we have like 2x B200. I think we have 2x H100 and like smaller variants. Help me understand the difference between AWS Lambda and Lambda. Lambda is, we'll see if I'll get fired at work. Formerly Lambda Labs, formerly, and then we just kept Lambda, like the meme from The Social Network. It's shorter. And so AWS Lambda is more a serving infrastructure for like gateways and stuff, whereas Lambda is a hyperscaler.

Serverless kind of thing. Whereas I think Lambda is like a VPS. You probably have serverless too, but you get like a box, can SSH into this box. It's like persistent. It's like the main queue that I've been using.

I will say most of my stuff, if I need stuff like that, I have a REST, I have like 7 Raspberry Pis upstairs that I use as my own VPS.

And you just hit the Lambda APIs with those and like create your own?

Yeah, like I'll just hit my own, the Lambda API instead of like hit the API directly and have my own service gateway and do all this crazy routing nonsense to like share the models out as I need to. So I'll refer back to the earlier discussion I said about when it makes sense to buy GPUs yourself. In general, the napkin math is you want to deploy a 1 trillion parameter model on a B2 H100 node and then scale down or scale up because you don't really want to have multiple nodes because it's not economic at that point. If you're like debating between, you know, 2 nodes of H100 to run something versus 2 nodes of B200 to run something, the cost of the B200 node needs to be less than double the price of 2 H100 nodes for it to start making sense. And like, that's not the case. H100 nodes are like 10 to 20% cheaper than B200. And so you want the biggest GPU because say it's A, time is money when you start renting hardware, and so you don't want it to be slow. And so you should pick the best one. And also models get optimized for the latest hardware now. So like if I do Codex, so like I can do /model and then you have this matrix. So like I'll do Luna xHi. Some of the models have been post-trained on traces that do well with custom tools like harnesses with cloud code. Any thoughts on how this affects new harnesses like the one you mentioned? It's a good pattern to do cuz this is what xJDR did. For a while and like for us at home where we want to like run something small and have it optimized for a harness, but also comes with the understanding of you are relying on a larger model to support the harness at this particular moment in this particular time. So using it to like control Claude code, out of the question, but using say KIMI to get traces to run open code, then that's great cuz you can pin open code and like have open code just be that or Py or whatever. It's good, but only when you actually make sure that it's sustainable for you and you're not like, the next update of Claude code breaks everything because they tweaked something. And this happens very regularly. Same with Codex.

I use this coding agent called AMP. They do this model routing for you. Like they have these agent modes that you can choose from and they basically say, okay, like your main agent, like if you use on low, for example, like it'll use GLM 5.2 for the workhorse and then it'll use SOL high for the oracle. The Oracle is like if there's a hard problem, it'll like route it automatically to the Oracle. You can just say use the Oracle if you're stuck. And I use this a lot and I found that surprisingly this particular coding harness, they actually, you pay API prices. So you would think it would be astronomically expensive. It's not that expensive as you would think. And what they do is they kind of tinker with it all the time. They have like a very opinionated take and they just say, okay, like this is a good model for this kind of setup. And they just like swap it in. Like right now they have Fable. I'm pretty sure this Fable is gonna become Opus 5.0. You can see like most people are using this medium version. This is like what is like everyone using. Looks like they're using like something a little bit different. They have this thing called the Puck. The Puck is pretty cool. It's just like, it's like an agent over the agents. You can say like things like, look at my threads and do something, manage my threads. Kinda like you can in Codex.

Yes, I could probably have Codex pack-in support for it, but like, I wanna do the fancy thing with the fancy thing the tool was used for and the fancy app that all of my cool friends are using.

Okay. I've used Cowork. I've used Codex. Codex can do everything Cowork can, and I feel like it's a lot better. Like the computer use in Codex is at a totally different level. Like the polish of it, you know, basically you can tag any application directly that's running on your Mac, like any application, and you can have Codex like directly operate it, which is essentially like what Cowork is. Another thing that really great about Codex is like it's mobile support. So you might have a laptop, you might have a server. Highly recommend having a computer you can remote into if you're using AI because you don't want to just have it shut down when you close your laptop. And so just having the ability to like look at all running sessions from your phone is like unmatched with Codex. Codex, like it just has like really good UI. It like shows all the computers running Codex, all the sessions within them. It tells you like what's running and you can start something on your laptop or computer and then you can just keep going on your phone. It's totally amazing.

Trying to get everything hooked up to Tailscale and then running Termius on your phone to SSH into your computers remotely to do the thing for some reason.

Because Termius on your phone, it's okay. But then the ergonomics are not completely dialed in. Like the keyboard is not completely dialed in all of the little nuances. So like there's a lot of UI polish in Codex. Like little things like diffs and steering or queuing and approvals and all this stuff is like very beautiful. And it's all rendered in the mobile app. Like everything is fully like 100% supported. Like, you know, and so like, that's really good. And another thing that Codex does is like Codex can control Codex. So Codex desktop, you can say, fan out, I wanted to do AI optimization of my website to optimize it for search engines. And it had like a list of 15 different tasks. But I didn't want it to just do it autonomously. I wanted to like investigate every task and have me look at every task. So I just said like, hey, spin up a new thread for every task, a new thread for every task, and put on my sidebar. I had like 12 or 15 things I could like go through and like learn. But then also like I told it, you can have threads communicate to each other. He's like, hey, this thread is running. You can just tag it by name, say like @name of the thread on your sidebar. You say, hey, like this thread did this, like can you tell that thread about what you did, blah blah. And it communicates each other. You can tell another thread to like, hey, like, look at all the threads and tell me, like, orchestrate them, whatever. It's, it's crazy. Like, you can get really intense with it.

I did experience that as well on mobile with Termius, and so I ended up like using my iPad more. Like, if I went to the gym and I wanted to work, I would use my iPad to like kick off deep research and like the model benchmarks, and then I just like clean up and write the draft. Everything else is like Codex goes brrr, Claude goes brrr, and does everything for me. It will be nice to be able to have like everything be centralized in a way.

It was really good having you. It was A lot of fun. Thanks so much, man.

Thank you guys so much. I appreciate you having me.