Building custom eval tools with coding agents w/ Isaac Flath

for some context, last time I— last cohort, I did a talk about FastHTML and HTMX, how to use FastHTML to build an annotation app. This time I'm building the same annotation app. I'm doing the same thing, but I'm showing it from a different perspective. Instead of like hand-building it and how it all works, I'm doing it from the perspective of how would I use AI to help me and do— build the same thing. So just a different style of working and Ideally, you have a little bit of both. Part of why I'm doing it this way this time is because the last stuff is publicly available. You can go watch the old talk. You can go look at the code. You can look at all the stuff. All right, cool.

Yeah.

As I said, AI-powered fast HTML development. I used mostly Cloud Code for this. What you'll see, I'll jump to the end here, is if you were in the last cohort and you saw it or you watched the recording of it. We end this talk with basically the same app. We still have this view of all the queries to annotate and all the documents. And so let's start and talk about what that is. This video, I go through actually coding it live, start to finish. And it is embedded in the talk, which will be public. So feel free to watch that if you want to see it like continuous, but I'm going to talk about some of the details. This is a Substack that Eleanor and I create. We are sharing a lot of free public content here. There's all kinds of stuff such as FAQs that we're working on, which are commonly asked about AI coding. There's tips that we're building out. There's gonna be information about all of our talks, which we have a lot coming up. Guides, reviews of different tools, all kinds of stuff, and information about our AI coding course. If you. Are a student of Hamel Husain's course, which I think everyone here is, you get an extra big discount than we're sharing anywhere else. In this, we cover everything about, a little bit about us and what we've done, the kinds of topics we've covered, the learning outcomes, all that kind of stuff. We link to a lot of stuff. I'd recommend following this at the very least, this eliteaiassistedcoding.dev. You'll get lots of free material straight to your inbox or straight to your Substack, however you want to consume it.

Okay, so you got Gemini, AMP, Cursor, and Open Hands already.

Wow.

Okay. So these are things I always, I kind of fiddle with them, but in the limit, I don't have time to try all of them. Like I haven't properly tried Gemini CLI. I've used AMP a lot, Cursor, obviously. I've also used Open Hands from time to time, but it's like, where do you see this going? So, okay, this is really seems helpful. It seems like you're going to be saving people lots of time by giving these like reviews. Um, I feel like people want this for eval tools, but we haven't gone there. Uh, what are you gonna do with, let's say, when you do more of these? I know you're doing more. Do you get a— are you getting a sense of some tools are better for some use cases in some situations? Like, how opinionated can you get with your review?

Yeah, absolutely. And let me just talk about AMP and Gemini since those are the top two. So there's something, I think, a little bit different about these reviews than you usually see from reviews. In that I do talk about my opinions about the tools and what I think it's good for and my opinions on the tools. But it's not just that. I actually do something. In this case, this is an actual open source task. Like it's an actual task that was actually merged into a real library. And I go through every step, what went well, what went poorly. There's a video, but there's also a text version if you just wanna kind of skim. Even to the point of, hey, here's some diffs that I needed to do to go in manually before I felt comfortable with the content. And it's a little bit different, I think, from a lot of tool reviews because it's focused on actually using it to complete a task and not just like a toy task. Hey, I built a to-do app, which is helpful and we'll, I'll do some of those for some things, but I'm really trying to review them in the context of an actual task that I actually need to do. So that's the first thing. And so like for, that's there for Gemini, for example, I go through, you can see it's the same thing. I've got diffs, I've got all kinds of stuff, screenshots of everything I'm doing. And you see here in here, I say things like, hey, I generated code for the desktop application. It was nice, but I don't think it's better than Claude code. That's there. And I say, however, I found the Gemini CLI super useful for multimodal tasks. And if you're like, I'd like to know what he means by that, all the information's— you could see exactly what I did and how I tested it and why I came to that conclusion. And things like content creation and writing, I like it for this. Yeah, so I have lots of details here that you can go through, or you can just look at the top and the bottoms to get my opinions if you don't wanna read through 'em.

But like, I don't wanna snipe you too much by this question, but how do you create these annotated blog posts? Like you have the screenshots and you have, do you do this like all manually? Like you go and you take screenshots yourself and you hunt and peck through it? Like what is going on?

No, absolutely not. So as I'm doing an AI coding course, I think there's best practices on how to use AI to do lots of things. And I have a programmatic approach. It's obviously human in the loop because I don't think you can just outsource everything to AI. But basically I built a small web app. That I give it a YouTube video and it will download that YouTube video locally just because I find it easier to work with that way. It creates a transcript. From that transcript, it creates like a content plan because the way we talk is sometimes the right order for writing, but sometimes not. Like the little bit at the beginning, for example, where I talked about— I talked about what I was going to talk about for for the AI coding of a FastHTML app. And now we're talking about this topic and we'll get back to the FastHTML agentic coding part. Like that works really great as a dialogue. That's how it works. Like we influence each other and that makes things a lot more helpful verbally, but in a blog post format, we might want those as separate sections. And so then it creates the content plan and it also creates screenshots of the YouTube video, and there's a config I can set however long. In this case, I set it for every 5 seconds, so I automatically get a screenshot every 5 seconds through the entire video. And then it creates this annotated blog post. And I'm using Gemini for a lot of that cuz I think it's pretty good for creating content. And what this is, a Markdown file, and it embeds what it thinks is the right image, the right timestamp to go with that content here. Now that's helpful. And it knows that because the transcript isn't just a text transcript, it also has timestamps embedded throughout. So it knows what I'm saying and approximately what time I'm saying it. And so with that transcript, with timestamps and the screenshots, it creates a Markdown file with this. And then my app, I go through, which is basically like a Markdown renderer. And I can—

is this, what is it built in? Is it also FastHTML since you're talking about FastHTML? You may not even get to talk to FastHTML cuz we might have, uh, took you way off track.

I know this is a talk about FastHTML, but that one is actually built in Air, which is a, an even newer library. FastHTML is a new web dev library. That's really great. You can build fast fat really quickly in it. Air is even newer that Danny is working on. Uh, just did a conversation with him about that yesterday. If you follow the Substack, you will get the recording and information about that in your email, but it's built in that. Which is, yeah, another Python web development framework.

So this writing app, is that something you might talk about one day, show people what it looks like or—

Yeah, I could. I'm happy to, I guess I, I build these internal tools that I'm like, is anyone interested in these tools?

I think those are the most interesting things, internal tools. So that's like what you're using every day.

It's true. Yeah. So I will talk about that someday. Record a video about it, and then I'll use the tool to make a blog post about the tool.

Yeah, very meta. Yeah.

That'd be great. Yeah. And then I think I'm just diving as deep as I can. So it starts with these tool reviews, the AMP one. There was some information about it that I liked, and so then I dug a little deeper. I will give you guys—

I like that you really built out the Substack. This is like very extensive Substack. It has like different sections. For like things, tips, tool reviews. You got your talks on here. You basically got like everything on here.

Yeah. If you see anything that's like really well organized, that's probably thanks to Eleanor's influence. So thanks to her for that. So that was a contribution to her. But yeah, and so like I'm really digging deep onto this. Like one thing, I'll show you a preview of something that's not public yet.

My 44 subscribers. That's amazing. That's great.

We started it like, what, like 4 or 5 days ago? So it's growing quick. So I would recommend.

Okay.

So this is, let's see here. This is a draft post. So which one is it? Or is it actually posted live? Okay. Anyway, I can't show you that, but I don't know where it is. But the, the idea is that once I did this tool review, I continued to dive into it. And so then I compared it to Cloud Code, cause I was like, okay, it felt nice. How do I do this? And this comparison, I've got a table. There's a preview of it on, on, on Twitter where some of it's blurred out, but the idea is okay, how does it stack up? And going into the details of, okay, when I deploy, you want a health check route. So that, you know, if your deployment is still running. Okay. So how did Cloud Code implement it? How did AMP implement it? Were they both functional? Was one buggy and one not? Which did I prefer? For example, there was other ones where I looked at how it implemented in GitHub Actions. Did it recommend improved approaches or did it just— that I liked or did it make— did it follow my instructions? And then also things like. Hey, in this case there was 20 lines of CSS and in one it embedded it into a Jinja template. And in one it created a separate CSS file and imported it. Both entirely functional, completely the same. In, in terms of me, I think having it in a separate CSS file is a better approach. And so I'm analyzing like which tasks did it succeed and fail, which tasks are better code, more organized code and going in there. And then Thursday I'm actually going to be. Talking to the CEO of Sourcegraph, Quinn, who creates the, the AMP agent to get even more information that you can't get publicly because—

so what I'm hearing is, okay, if you're using coding agents or assistants to help you code or really help you work at all, then you should pay attention to what you're doing because you're like, you're like dedicated yourself, as far as I can see, you dedicated yourself to reviewing all the tools deeply. In a very unbiased, let's say, like you, you don't work for any one of these vendors. You're like just a neutral third party.

You just— I don't work for them. I don't get credits from them. I don't like, there's nothing. Yeah.

Okay. That's great. Yeah. This is super helpful.

And you can see right here, there's 9 published cuz we started this like last week, but there's 14 drafts. So those are all basically written. So there's a lot of good stuff coming. If you sign up.

Yeah. That's— and the, if I recall correctly, like you're going to do this deep dive of let's say AMP versus Cursor, and it's not just, you're going to have all these tasks you like compare them against. There's like a teaser that you showed. So that's the type of analysis that you do, I think.

Yeah. Actually, I just found this. Yeah. So like in this case, this is just like my notes. This is a screenshot from Google Docs. Cause I'm like, I'm like, actually, okay.

So you're showing us a preview of a post that hasn't been set it's not out yet, so you're giving a special preview at this point.

That's right. Yeah. I haven't published this yet. I'll publish it at some point, but yeah. So this is like just for, and this is just for infrastructure. This is a post that dives into just how Claude Code versus AMP on infrastructure tasks specifically. And there'll be other, there's other domains like multimodal and other things. Like multimodal, I think comparing AMP to Gemini makes more sense cuz Gemini's like the king of multimodal. But for this, for Claude Code, I think it does really well. And it's like, how did the health check? Work. In this case, oh, the CI here, check to this deployment worked. And so this was really like building an app, Dockerizing it, sending it to fly.io, how much memory was allocated. Like in this case, they both worked, but Cloud Code overallocated the amount of memory, which is a little bit more expensive that wasn't needed for this task. So quite a lot of detail that I'm going into here.

That's really cool. Thanks for sharing that. Sorry for taking you off topic. I think that's super interesting, this whole kind of journey that you're on. These are basically like the tools people are using more than anything else in AI at the moment, as far as I can tell.

Yeah, yeah. And I think it's, it's important because these tools can be expensive to try them all because it's time-consuming. If you just do the free tier, you don't get the best models, and that might mean that really great tool with a bad model just doesn't feel very good. And I think we're not just reviewing tools, we're also giving guidance on you should use it for this task versus that task. And here's how you create context, which you'll see some of that here. So yeah, I think it's when I see how people use AI for coding, almost everybody.

And why are these slides look so nice? Like how did you make this?

Oh, I used AI to make it.

Okay.

So like AI coding practices.

Okay.

Yes.

So it'd be like, is it using Quarto or like what?

No, I had it make it with HTML, CSS, and CSS. AI is very good at, you know, it's seen a lot of HTML. There's a lot of that around on the web. Same with JavaScript and HTML. And so yeah, I'll just actually just show it to you. So if we— oops, click. If you go here, this is the presentation right here. Wow, this is like left and right. That's JavaScript.

So this is HTML and JavaScript. Wow. And you just had some—

you just had the model, you told it what you wanted to say, and told it what I wanted to say, the style I wanted to do, where I wanted to put things. I could go in here, I can go in here. If I, if I say, oh, I don't want this title, AI-powered fast HTML development, I can easily go in with my editor and cool, I can edit these titles. Like I can do it, I can find any text I wanna change easily enough. I can see that this is a slide. It looks like a lot of HTML, but at the same time it's all the contents here. And I let it handle the styling and the JavaScript and all that.

How did you find this compared to using Google Slides? Was it a lot harder to prompt?

It's a lot easier in my opinion. Like, I, I honestly, I just, I, it, the styling aspects probably took, I don't know, 2 or 3 prompts. Like, I asked it to create it and then I was like, it looks ugly, can you do it again? And it made it better. And then I said, hey, I want my website in the top corner. I didn't even tell it to make the lightning bolt, but it put it there and I was like, I like that.

I think people would like to— do you save your prompt, your history? Did you use Spec Story when you did this, by any chance?

I think I did. Yeah, I think I do have it.

I can—

if I don't, then I can make a new one and share it. Yeah. So yeah, if that's interesting, I guess that's really interesting.

It's like maybe I should create custom presentations now. This is— this looks so nice. So yeah.

Yeah. It's, it's really great. I like doing it. I don't— I hate fiddling with like PowerPoint or Google Slides and stuff. And then let me. Screenshot this and move it over. This just feels so much easier and nicer. Even before these AI tools were this good, I didn't use PowerPoint. I would just use Quarto for everything. And this just makes it even easier for me than Quarto, cuz I don't even have to, like, I don't need to think about the callout boxes and stuff.

Interesting.

This video is about 12 minutes long. It walks through everything. I'm not going to play it here. I wasn't planning to, but if you're like, I want to see it in action and what he's actually talking about being used, that video is embedded in the presentation.

What do you build in that presentation?

I build— I build— hang on a sec, I got screenshots here.

Okay.

Yeah, I build this RAG annotation app. So these are queries. Some of them are like PDF documents, some are long form, some are short queries, and they annotate for this query that the user entered, was this good or bad, relevant or not relevant? And they have a small notebook.

So it's like an annotation app like we teach in the course.

So yeah, yeah, I would say that it is a little bit more elevated than vibe coding, but basically—

oh yeah, it's definitely more elevated, but I've seen you, the way you do it is definitely pretty good.

Yeah, this is this RAG thing. The challenge is like there's 7,000 records, you have one query, and then the annotator needs to say for 400 records, is this relevant or non-relevant? It's got to be like fast, it's got to be like convenient and easy, because going through 7,000 records is painful and it just won't happen if it's not convenient. All the same stuff Hamel Husain talks about in the course, it needs to be really low friction. And there's several steps I did here. I created a Ruler context file, which we'll talk about, created a plan for spec-driven development, generated it with Cloud Code, and then Cloud Code was able to test with an MCP server I created that helps me with web development. So let's go through each of those. Ruler is something that Eleanor created, my co-instructor of the course, and it is— allows you to create a.ruler file folder and inside it you can put any context for AI you want, any Markdown file. So in this case I've got UI component guidelines for Monster UI and fast HTML patterns, two Markdown files. And the cool thing is, is it auto distributes any agent. So if you watch here, I can list the directory and you can see all there is a.ruler. You hit ruler apply and it applies it to Cursor, Windsurf Client, Firebase Studio, Open Hands, Gemini. Cli, and if you, we list it again, you can see it's got all the right configs in the right format for windsurf and cursor for all of them.

And now you just maintain it in your.ruler file and it basically compiles to all of this.

That's right, exactly. Now you can have a config that says I only want to export it to, say, cursor and Cloud Code or just Gemini CLI and OpenHands. You can create that as a config or you can pass it as arguments to the CLI so you can control it yourself. But this is important because it lets you create an LLM context and then port it to any model. And so like when you try a new tool, you're trying a tool, but you're not trying it as an, I don't have the right context and I'm, I don't, you're trying it as you would really because you have all the right context built in. And so you're seeing how it would really work in practice much easier rather than trying to figure out what all the config files are.

Yeah, great. Because like at first I was like a bit annoyed by Lockin. I was like, oh, I have all these like cursor rules and I don't feel like I can change. And then I started to do something janky, which is like I would have my agent.md files or curse the Open Hands one. I would have it all refer to the cursor rules. Oh, go look in the cursor rules. But for some reason that doesn't feel, doesn't work as well as the native stuff. So this seems like the best way, like you could just have all the native stuff for whatever tool you want.

And where I think this really gets even beyond that, that I think a lot of people miss out on on Ruler, is that this is really good for like teams and like work settings because everyone has their own opinion about what tool they want. Tell everybody they've gotta use Vim or tell everyone they gotta use VS Code, they'll be like a riot. It's the same thing with AI tools. Tell everyone you can't use AMP or you can't use Cloud. Like some people like Cloud, some people like AMP, some people like Gemini CLI, some people like, there's lots of tools. And people have different tastes and preferences. And now with those ruler files, you can have the entire team maintaining a good context for how the standards for your project, how your project works, the kinds of internal libraries, and people can port them so everyone can maintain the same shared context, but port them to whatever tool they're most comfortable working in. And so I think that's really powerful as well.

That's cool.

When you're doing a lot of stuff, it's good to start with a clear plan. Tell the AI, so spec-driven development, tell the AI what you want. You can type it out. A really good way is to voice transcribe it. These agents will give you a plan. I find that it's best to ask it to write it in a Markdown file.

So I have a question about this. So I agree with you with the spec-driven development, create a plan.md first and all that. You know, and so you can review it. A lot of agents like Claude and AMP will have a to-do list or create one like a plan. Maybe even AMP does like a plan also. How would you say this, what you're saying is different than that?

So it's not really different. So depending on the size, you can have a plan and then you have the agent act on the plan. If it's something that I think is really complicated or tricky, or I wanna be really hands-on, I actually create the plan. File, and then I have to create a to-do file, a to-do Markdown file, and then I work through those to-dos. Now, the reason I say it's nice to do it in a Markdown file and have it write it externally rather than in the agent is because like when you're having the AI create the code, the one thing that you can do to increase the quality and the consistency, which the consistency is like most important, cuz if you don't have consistency, you can never build intuition. You know, about what'll work and what won't. And by creating these Markdown files, you're putting it in a format that you can edit quite nicely. For example, here it's just helping me create the plan by looking through the database schema and all that. So I didn't have to, but then here you get this plan in this Markdown file. They will, these models will create a plan and even a to-do list or both, but going through and reading this and saying, oh, this current eval type. Like, why do I have this as a separate feature from a button? The button should just be highlighted if it's good and not highlighted if it's not. Like, which button is active? And so I'm here editing, saying, no, this is what I want. That's not quite what I want. This is completely off base.

And from here, they just like type in their thing to the prompt and just like we hope and pray that it works. And I think I've seen you be really successful with a lot of vibe coding pro— or maybe I shouldn't call it vibe coding anymore. AI coding projects is you do this, like you're really careful about the plan. Is it good?

Yeah. And I think the really critical thing about it is that partially you're going to have more success. You're going to be able to complete more tasks with AI, which is helpful. But I think the really important thing is you're going to be able to, you're going to get more consistency over what works and what doesn't. And I think that's why a lot of people end up with. Finding that they're not as productive with AI because something— AI can't do everything. But if you're not consistent and you don't get consistent results, then you end up in this, yeah, might as well try it. And then you go down a rabbit hole and you don't know why. And then other times it feels like it's magic and everything just worked. It's cool. I built this cool interface. And so I think it's, you'll have more success if you spend the time to understand and define what you want the AI to do. And you can specify when you want it to stop, specify how you want it to test things. And then you'll also be more consistent and like, you need to be able to see the patterns of what's working and what's not, if you're gonna be getting better at any skill. Right. So yeah, I think everyone knows this is a good thing, but I think it's not done because people don't know how to do it efficiently. I feel like it's like a huge burden, but I honestly, I spend 3 to 4 minutes on this. If you know what to look for, you can go through it very quickly. I know our time— do we have enough time for one more topic? Yeah.

Okay.

So the last thing is I built a little MCP server and this is where like you can add more context to models that you cannot get through static context alone. So this MCP server has things to start my local app. It can navigate to pages and routes, it can click on elements. It can interact with my app. This is using like something called Playwright. And so just like you create an app and you think it works, like how do you test it? Like you typically, you open it up and then you like click around and you like type something in and you save and then you refresh the page, see if it's saved. You look at the database. Well, your AI agent should be able to do all that too. That's all stuff, you know, that's gonna happen. And the interesting thing here is it can also get console logs. So when you develop a web app, You get all the server logs streamed to your terminal. Usually you can redirect those to like a file and have AI look at that file. And that's a good thing to do. What you can't get is the actual browser console logs that happen client side. So these are like HTMX errors. I know I'm getting a little bit in the weeds of the information, but JavaScript stuff do not show up in your browser, in your server logs. You have to use something like Playwright in order to get that information. And so what that means is your model just doesn't have those error traces. And usually what that means is you just click around yourself, copy the error, paste it back, wait for it to do it, click around, copy that error, paste it back into the context. Then again, that's just, you can do it. It's just slow and inefficient.

So this MCP basically can drive a browser and navigate to your site and whatnot.

Yeah, exactly. And you can see this agent doing this exactly. It says, oh, navigate to this path. And this is an ID path from a thing in the database.

I have a question about this. When you're using this MCP, how does, how does the Playwright MCP know like what port your application is serving on? Is it, can it see the terminal also? Can it see like your main terminal where the server, where your application is running?

So I actually created my own MCP server just for this one project.

Oh, okay.

Thankfully it's pretty quick to do because you can use AI coding practices to make it. Yeah. I know I'm beating on that, but so this MCP server I created this and this isn't a general Playwright server. This is just a server that knows how to run this main.py file specifically, do it on a port and do all that. It was helpful for this project. Do you usually do that?

Do you create MCP per, like your own, like internal MCP for debugging? What does that mean?

If I find myself copying and pasting something more than twice, then I know I gotta have, if I'm like, if I I'm copy and pasting anything into the model more than once, basically, I know I either need to fix and improve my context file. So I don't end up with this like inefficient, like PaperCut thing, or I need to make some sort of MCP very quickly that gets that context. And I think it's, it's always sounds like a burden to people. I think, oh, that's a lot of time wasted, but I build web apps all the time. And like now I have this MCP server.

It would be a little bit more intuition, like why? Because I know that there's MC— there's play— definitely Playwright MCPs that you can find, and there's like popular ones. What is your intuition? Hey, let me just create my own.

I didn't think this would take very long, and I don't want to look—

and it didn't take long, you're saying?

No, it took me 5 or 10 minutes to make it work. If I thought it was going to take me a few hours, I would probably have looked to see is there a solution out there that just works, but I figured it'd just take a few minutes and it'd probably take me less time to make it than to learn some complicated AI product docs, cuz they're not always very good. And that's what I did. And I think it's like, often people don't invest this cuz they often think of, why am I gonna spend an hour for something I can build in 2 hours? And that would be true if the only FastHTML app you're ever gonna make is this one. That seems like bad math. You're making a lot more, but Every— now that you've made this once, every FastHTML project you ever do, you can just copy this mcpserver.py, copy this over. You can either make it more general or you can just copy it and be like, okay, this works as long as I've got a main.py file in there. I can use this mcpserver on any FastHTML app. Same with all the contexts. If I create the context, I can do this. And for most projects, most significant projects, it's not, oh man, I gotta make context for Django. That seems like a lot. It's like you're probably working on Django on a commercial project and you're probably gonna be using Django for like months or years. So I guess in 5 or 6 hours— I have a question.

You may have, uh, covered this in your lesson with Danny, but what do you do with idiosyncratic projects like FastHTML, Air, whatever? These are like new libraries where the LLM hasn't seen seen much of it. You can pull the documentation with Cursor because they have the docs indexing, but it's hit or miss sometimes in like how good it's able to fetch those docs. What do you do like across these tools? Like, what have you found is the best? Do you just say look up the documentation, let it do its thing, or do you be more intentional?

No, I do something more intentional. So there's a guide on the Substack— another reason you should subscribe— but Um, there's a couple of planted questions, by the way.

I'm just reacting.

We didn't talk at all. It feels like that. Cause every time you've got a question, I'm like, I have an answer. It's written.

Okay.

Anyway. So yeah, so there's levels of context. I'll cover this quickly, but basically there's like general levels. This is, and this was actually stuff for Hamel Husain's course last time around. There's stuff like Repo Mix and PasteMax and Git MCP. And these are all different approaches of automated ways to get context that works on almost any library. And they're a good start and there's like chat interfaces and stuff. So those are a good start. If I don't know if I'm even gonna use it and I don't wanna spend any time on it, I start there. Eventually you can go down to tell it to grab the, you can either tell it, hey, go look at the docs, or you can tell it to find an MCP created by that client. Like for example, Braintrust has an MCP, whereas Phoenix and Inspect use the lms.txt format, which you can turn into an MCP either with Hamel Husain's tool or LangChain's tool. Or grab them. So I cover like these different approaches, but ultimately, like, if you're gonna be working with a project for like months, it's worth the time to spend 2 or 3 or 4 hours to get like better support from your LLM. It's like if you could type 10% faster and it would just take you 3 hours of practice, like you should probably do it. Cause like you type all the time. And this goes into the approaches that I do. So I start with GenAI to get something into a Markdown format in some way. Or copying the outer HTML if I need to. And I get this context for the docs specifically that I need. And over time, I know this looks like I do it all up front, but over time I personalize it. So I say, for example, this says in the docs, it says, hey, you can do this or this. And I say, for my project, I do this. So I edit it.

So you're like editing those, like these concatenated docs that you assemble, or do you even concatenate them? Do you just put 'em in a directory?

I just put 'em in directory, separate Markdown files, put 'em in your ruler directory. And yeah, just when something is like, why did it do that? The answer is usually like your context wasn't quite right and you can just tweak it. And then every time you have that same thing come up again, it'll just be right and be opinionated.

So what about, are you afraid that, let's say, okay, so you're showing Braintrust here, right? Like how you made a context. To help you with Braintrust. Are you like worried that the Braintrust library is gonna change so fast that your things are gonna, your customization is gonna get way out of date? Or you're not worried about that?

So I'm not too worried about that for a couple of reasons. One, there are ways to automate some of this process. I'm showing the manual approach, which I think you need to understand first, but there's tools like Spec Story has a tool, others have tools where you can have it automatically update and tell you when your context should be changed. And you can record your chat history and easily analyze your chat history to figure out these changes. The other thing is—

wait, what is that? So what do you mean?

There's like automatic rule derivation stuff.

Cursor. Okay. We saw that you complained about XYZ. Do you wanna add a rule? Like that kind of thing?

Yeah, there's that kind of thing. Spec Story's got one, but there's also like the Spec Story, you get all your chat history and markdown files, which is really convenient because you can ask any agent. Claude or AMP or Gemini for this, but if a repo changes and you have really great context that's in the specific format that you want and you say, hey, this library changed their API completely, here's the new datasets thing, go update this file with the new API. It's very seamless to do that if there's massive breaking changes because it already sees—

So you'd be teaching how to use Spec Story or whatever, cuz honestly I have not, I know our friend works on it, Greg, but I haven't gone too deep in it myself.

Yeah. Yeah. We will be actually, we are scheduling a talk for them to talk about some of the product-driven development stuff in there as well. And I think there'll be more information about, about Spec Story coming as well. And we'll cover that, that stuff in our, on our course as well. Everything you need to know will be in the course for sure. And yeah. And so if you have something and you're like, hey, I have this context, but it's outta sync because there's these major breaking changes, that's the kind of text task that a model can do really well. You already have all the information, you already have it in the right context, and you're saying like, just tweak it for the new API. Now, if you're using Braintrust or Phoenix or whatever and they change their datasets API completely, it's probably not that they don't have a datasets API anymore. They will always have a datasets API. That's one of their whole things. And if you have good context, getting it updated for a new API is was not super time intensive or challenging. And you need to invest a little bit of time anyway, cuz your code probably all broke too.

That makes sense.

So yeah, so in this case it navigated to a route, it clicked on an element, it got a 404 error. It's checking the definition. It's looking through the server logs with its tool. It's, it's finding it. It did check the browser console and it did that loop without me having to like manually navigate. Now when it was done, I did manually navigate. And looked for the error, looked for errors and tried to fix it. But it can catch things without me having to like copy and paste it back or say, okay, it didn't work. It's got those tools and I add the right context to instruct it to, to try those things.

Yeah, looks really good. As usual, I'm, I am fascinated by all the meta stuff that's going on in this presentation as well as the presentation itself. So sorry for taking you off track.

Oh, it's great. I love talking about it.

But yeah, thanks a lot, Isaac. Appreciate it.

Yeah. And Substack's there. The course is here. There's free talks you can jump to. If you go here, this is a link to the course, but all these Lightning lessons are free from like parallel agents, Gentec UIs for— universal UIs for Gentec coding is gonna be great. There's ones on AMP for like tool reviews. There's one tomorrow, which is building—

Oh, tomorrow's a spec story thing. That's interesting.

Yeah, building a product video in a day. And this is from someone not with a video production background. Like, I think a lot of times when there's like videos out, you're like, hey, look, this like professional video editor used AI.

That video is impressive. Like, we should— this one, I should share the link to that in the chat. Let me get it.

Yeah, and this one is built in 2 hours from someone who doesn't have that background and doesn't even really have a strong coding background, using like ReMotion and like all that stuff. So it's It'll be really interesting. I think everyone would really enjoy. So that one's tomorrow. But there's a whole bunch of ones, like the chief architect of OpenHands is gonna be giving a talk. So if you're interested in like how these like agents work, that's good. OpenHands is like an open source competitor to Devin, which is super cool cuz the chief architect can come tell you how it works. If you want to dive deeper, you can just go look at the code cuz it's open source. Is Vibe Coding a Broadcasting Graphics Solution? Steffi from EveryWow is gonna be giving that talk. Yeah. All kinds of stuff.

Which one's the, the one from EveryWow? What is that one about? Sorry.

Yeah, so she Vibe Coded a broadcasting graphics solution for yeah, actual, an actual broadcast.

Oh, wow.

And made some throwaway software for creating a dashboard to do it. And like, this company makes incredible videos and incredible content. And so again, it's— I think the theme here is like, I'm showing you things that people are doing that are actual commercial value, and they have commercial value because they were used commercially. These aren't like, like toy demos.

Yeah, yeah, they're not hype demos for Twitter. They're like people using it in their business, like legitimately, right?

Great.

Okay. This is cool. This is a lot of, a lot of stuff going on. So that's yeah.

Yeah. So as a reminder, just check your email. Hamel sent you a discount code, a bigger discount code than you'll see in the presentation or bigger discount code than I'm posting on Twitter or anything. So check that out if you're interested. That'll probably be active for 2 more weeks before I close it down. But yeah, take some time to, to look around.

Cool. All right. Thanks a lot for coming on, Isaac.

All right. Thanks so much. See ya. Bye.