Building a Personal AI Model Map [AI Operators Bonus Episode]

Boot it up.

Lights on. Game time.

Come on.

I operators, we're building the future tonight.

Press start, watch the data take flight Flip that switch feel that code ignite let's plug this world into the.

Coffee.

On the desk Hair don't care.

All right friends, today we are going to do something a little bit different. One of the things that I was contemplating at the end of last year was actually doing a spin off podcast called AI Operators. The idea of AI Operators would basically to be what I was calling a skills cast, where instead of focusing on the AI news, it would be all about using AI tools, building AI projects, and ultimately what I decided is not not to do that per se. It's still something that's very much on the table. But where I wanted to start at first was to experiment with it in the context of the existing AI Daily Brief community. And the New Year's AI resolution kind of gave me the perfect opportunity for that. For those of you who haven't heard or watched it on New Year's Eve, for the last episode of 2025, I put together basically a self guided AI resolution program which was 10 weeks of projects that anyone could do to try to upgrade their AI skills heading into 2026. Now of course we weren't just going to leave that an episode, so I vibe coded up a website to go alongside it. The website has the program with all of the different resolutions and projects that you can do, as well as an ability to share what you did with the rest of the community. We've actually had literally hundreds of people share their projects already in just the first week. And part of what makes this so fun is that because this is all vibe coded and because candidly the stakes are fairly low, basically anytime that someone has had an idea to improve the experience, we've been able to just jump on it and do it. So for example, this team feature came when one of my patreons said, hey, I'd love to be able to do this with the team. Prompting me to think to myself, well that's just about the most obvious thing that I didn't think of. And so sure enough, I was able to push a teams update about 10 minutes later. And now in the past week We've had over 200 teams Sign up to do this together. By the way, big shout outs to the Google Cloud startups team, by far the biggest team in the group, they have had 90 people joining. We've also got a team from Meta that has about 35 and a ton of smaller company teams that have 5, 10, 15 people on it. So really, really fun to see what you guys are all doing. Now, coming back to this operator's bonus episode, though, we are now in week two model mapping. And the idea of this, in short, is to help people do a set of tests so that they start to get a feel for what models and tools they like for different use cases. One of the lowest hanging fruit sources of alpha in my estimation, for how to take more advantage of AI than most is to have this sort of personal map for what you think different tools are better or worse at. Now, with this, I'm not saying you have to go subscribe to the premium version of all these tools, but even knowing which among them are best at the free level gives you more power to use the best option for your particular use case at any given time. And so the idea of week two was to have people go choose a set of models, test the same prompt for a particular use case, and create a personal reference document for when to use each tool. And I think that that's cool, and I think that people who are doing that are getting a lot of value out of it. But I wanted to take it a step farther. One of the big shifts for me over the last couple of months, but especially coming back in 2026, is pretty much for everything that I'm doing, I'm asking myself, is there a way to build something, some software, some application that would actually make this better? And so I started brainstorming with Claude, thinking about what sort of software might be valuable as part of this model mapping experiment. What came out of that brainstorm was a recognition that actually, just from an information density perspective, testing a bunch of different use cases across a bunch of different models could get very unwieldy very fast. And maybe it would be valuable for people to have a place where they could contain all of that information, a home that would become the receptacle for what they were discovering as they were discovering it. And thus was born the model map builder application. Now, I'm using Lovable for this, but. But you could use replit, you could use CLAUDE code. And let me first give you a little tour of what I've built so far. So the model map is exactly what it sounds like. It's a place that should, once done, have a full map of your personalized AI model recommendations for different use cases. So what are the different components? The goal of this is not to actually do the tests in this. That's going to happen in the interfaces of the particular tools you're testing, but it's to have a place where all the information can live. So one part of this is a use case library. I wanted to help cut down on the blank slate problem. So I put together a set of about 20 different common use cases, split across different areas like strategic analysis, writing, visual design and code. Where if you're trying to do this test fast, instead of having to think up a use case, you could just go in and find one that's either interesting to you or as close to something you often do. For example, you could do a competitive landscape brief to understand competitors in your startup's area. You could try to automate a common workflow, or if you are a knowledge worker, you could do the single most common thing we do, which is make slides. Now, we also added the feature for people to be able to add use cases, because obviously these 20 aren't going to sum up everything that people want to do. So when this launches, people will be able to add their use case, they'll be able to add a title, give it a category, give it a description, share a prompt template, which of course other users don't have to copy, but which makes it easier, and then decide if this is just valuable for them or if they want to share it with the community. Over time, if lots of people do this, it'll become a use case repository that could be valuable even beyond this model mapping exercise. Now, with your use cases in hand, you go to the test lab and this is where you set up and structure the test that you're going to do. So let's choose presentation slides. It's going to pull up that prompt template, which we can copy and take over into whatever tools we're going to use. It's going to show us if we've run that test at all before, or more specifically, if we've added information about a test that we've previously run. And it's going to give us the ability to select our models or tools to test. Now, one of the decisions that I came across was how to break this apart, because the lines between model and tool get a little blurry. So, for example, 11Labs exists both as a tool, but also 11v3, the most recent model, is here in the models list. And so you can pick any combination of models and tools. We'll do GPT Image 1.5, Nanobananapro, and then we'll head on over to Tools and add Gamma, genspark and Manus. So now we've got our test all queued up and we can take that prompt, switch tabs, go over and do the test and then come back and report the results. The scoring system I've set up for now, which is of course like all things with this subject to change, is a 1 through 5 stars on accuracy and quality, a 1 through 5 stars on style and fit, a speed rating slow, medium or fast, and a three star X factor to try to be a catch all for things that I've either missed in the rating system or which are unique to the particular use case, or to simply give you a place to say there's nothing better or worse about image 1.5 versus nanobanana I just like it better. Once you're done you can save your results and then ultimately the models that you've rated live over in your My Models tab. Each model is going to have an overall rating, but it's also going to show the specific results for different use cases from different tests. You can also go back and see your history of tests where you can also edit the results if you want to change how you rated something. Now there is still a lot here that I want to change and tweak and one thing that I wanted to share related to all of this is this essay from Google's Senior AI Product Manager Shobham Sabhu, called the Modern AIPM in the Age of Agents. Now I actually read part of this in the normal long read Sunday episode tomorrow, so I'm not going to spend a ton of time on it, but this is the thing that I wanted to share of this mental model from handoffs to hands on. In the old model he writes, PMs figure out what to build, write the spec, the engineers build it, the PM reviews it, and that's the iteration cycle. In the new model, the PM figures out what to build, the PM builds it with agents, the PM evaluates it, iterates quickly, and when they like it, they hand off to engineers to go live in production. Now obviously for our purposes we're skipping that handoff to engineer phase, just going straight to live. But I think that that frame of reference is really important. So for example, before finishing this, I wanted to show you guys the latest thing that I was working on in the test lab. We're switching the system so that you can add a model and whenever we want to do something new and big. In Lovable at least, I suggest you use the chat mode, which is the planning mode, before you implement the plan. Lovable actually uses different models for the planning as it does for the execution. So in this case this morning I had noted that when you click on Manage while selecting models to test, it doesn't really do anything and the way that it's conceived doesn't make sense. So I wanted to replace it with an add button where you can add the name of a model or tool. The planner then goes and looks at the code and figures out what the issues are and then it identifies options. So in this case it discovered that adding models is only for admins, right now. So did I want option A, allow all users to add models or option B, keep that admin only? It assumed that based on what I had said, I wanted all users to be able to add their own models. And so it put together a plan to change it so that indeed all users could add models. However, that's not actually what I want. When a user adds a model, I only want it to add for them. I don't want it added to a global list. So we refine the plan and now I switch out of chat and say implement the plan. Now that the plan is approved, it's going to go off and do its thing. And usually this is the point at which I start thinking through the next thing that I want to fix, assuming this goes right. So now theoretically this is ready and thanks to the magic of editing, you didn't see that that took two or three minutes. So now we have this add function. We can add the name of the model provider, the type, whether it's a model or tool, and an optional description. Now, one thing that I haven't figured out yet is whether I want to add. Actually, you know what, let's do this. Just so you can see how a fast feature goes, I'm going to use Whisper Flow, so you can see how that works too. Holding down control and option. On my computer, when a user adds an option, adds a model option, that is give them the chance to have a checkbox at the bottom that says Suggest admins add to master list or Suggest Admins add to main list. So that can flag in the admin. So it can flag in the admin that this is a suggestion that we should consider for adding to the main list. And since this one doesn't really need all that much discussion, I'm actually not using the chat. I'm just going to go straight to updating it. So that edit took about one minute and let's see if it worked. Sure enough, we now have the Suggest admins add to the main list button. So that's mostly what I wanted to show you guys today. First of all, I wanted to give you a preview of the model map, which should already be released by the time that this comes out. I'll update aidbnewyear.com to have it, but I'll also put it in the AI Operators community and then from there I've got a few more things I want to do in the My Models page. I want it to be viewable by models or by use case. And of course I got to give it a branded domain anyways. Hopefully you now have a better sense of the model map tool, but also the emergent process by which I'm starting to find myself translating opportunities into software in a pretty consistent and ongoing basis. Since we have been fully back in work mode in 2026, I basically never don't have some vive coding project going and usually multiple at the same time. And even as someone who is incredibly deep in this space, it's taken me a year of some of these tools being available to really fully click into that way of thinking. I hope some of this helps you on your journey and I'll see you over in the AI Operators community.

Coffee on the desk and hair don't care.

Yeah.

We run it from the chair, chair chair bots on the left, charts on the right we tune up the models then we send them in to fight. One more tweak than a big deploy we break it, remake it like it's our favorite toy from hello world to.

Holy wow We're shipping tomorrow so we're shouting it now hey.

Hey operators we're building the future tonight.

Press start, watch the data take flight Flip that switch feel that cod ignite let's plug this world into the light.