Building Eval Tools with FastHTML

Okay, so just a quick introduction on Isaac. So Isaac is one of the most brilliant people that I've worked with over the years. I've, I've, uh, worked with them a lot in open source, um, you know, working on things like fast.ai and nbdev and all kinds of other stuff. Um, one of the things that Shreya and I always talk about as the most highest leverage things you can do is to build your own data annotation apps. You know, it's like, especially with AI, because AI is really good at building like data annotation apps. Um, and so one question that always comes up is like, okay, what tools do I use to write these data annotation apps? And like, how do I do it? You know, like there's a lot of different choices. Um, one, you know, I'm a Python developer myself, and so, uh, there's not you know, great tools per se with Python relative to like, you know, the JavaScript ecosystem, TypeScript. Um, and so that kind of, you know, can hinder data scientists even like, you know, if you don't want to use things like Streamlit or something, how can you use something like, like a more full fidelity? Um, and so Isaac has— is an expert with FastHTML and he's also been building a lot of annotation apps. FastHTML is a Python framework that he'll get into. But really, like, he's going to show you this framework. A lot of the principles and mindset will apply regardless of the stack they're using. But the reason I wanted to bring Isaac on here is he's been, he's been building a lot of data annotation apps. So I'll hand it over to Isaac.

Awesome, thanks. So I guess I'll jump into it. I'll be talking about the annotation apps. There's, uh, I put in the Discord for, um, this talk under guest speakers, you can find it under my name, um, the companion repo as well as a link to these slides, which is also in the companion repo. This is with Quarto. Um, that repo is going to have what we talked about today but a whole lot more. Um, it has the cursor rules files I used when I was making these example apps. It has a simpler version of the app. We're going to walk through some, um, code today of a simplified version of a real commercial use case. There's an even simpler version. There's also a FastAPI and React version of the app, so you can kind of compare FastHTML with potentially another approach. Wade, Wade helped me with that, so I really appreciate that. And if you have a good easy time running everything, that's probably also Wade. He did a lot of work on the READMEs and things. There's resources and links to learn FastHTML and MonsterUI. There's links to, you know, where you might deploy it now. So there's a lot of stuff in there. I don't have enough time to go over it all, but I wanted to share it, so I wanted to put it somewhere anyway. So we'll jump into it. Building annotation apps with FastHTML. So today— what did I draw on my screen? Okay. Today we'll, we're going to go over a real-world use case. This is not purely academic. We're, I'll show you some screenshots of the actual application that's used by the client, show you what FastHTML is super briefly, and then we're actually going to walk through code of a minimal version that works and runs. And again, all that code's in the companion repo so you can run it yourself. And then we'll do, talk about some of these resources and Q&A if we have time. So the actual commercial use case, this is for a company called AnkiHub. They work with Anki flashcards and they use it, you know, they use AI-powered tools for it. So this particular use case is for, you know, the medical domain. So a medical student may want to type in purine and get a list of flashcards that they should study related to that topic. Now where this gets a little bit tricky is because there's a lot of, lot of things they might query on, everything from a keyword to uploading a 50-page document. We'll talk about that more in a minute. So this app, I created it in collaboration with LabCodes, which is a consulting company as well. They're a Brazilian software studio that designs and develops high-quality web applications. I've got their link there. So if that's something you need help with, check out their website. And then me, I'm a FastHTML contributor. I've created Monstro UI, which is a UI framework for it, and I do independent consulting for, you know, products and companies that use AI. So let's get into the stuff that's more interesting. So this use case I mentioned, this is a real-world use case. You can go to ankihub.net and look at their site and look what they do. But the idea is these medical students want to find flashcards flashcards for their exam. So it is a retrieval task. And the real challenge is the queries are, are very, very variable. They might type in a couple of words, they might upload a chapter of their book, they might upload a PowerPoint presentation in PDF. And so really the inputs are really all over the place. Typical Retrieval tasks, the user enters a query, something like a Google search query, and they return documents that are very, very long or a chunk of a document. This is a little bit different than all the real ones. And so as with most cases, a standard out-of-the-box tool, there's some reason why it's not, you know, just plug and play with something. And in this case, it's because This is complete opposite. We might have queries that are pages and pages long and results are a sentence or two because they're flashcards designed for one specific fact. And so, and then the last thing is this is a technical medical domain, you know, very specific technical terms have very specific meanings. And so you need to match. Someone asks for a specific technical term, that, that's very important. It's— we don't need some kind of similar in the same domain. We, we have a very specific targeted meaning for that word. And so all these combined mean it's very difficult to just plug your data into a, you know, any of these tools and run it because You know, like with every product, if you could just plug and play it, it wouldn't, wouldn't be that valuable of a product. So cool. So this is the application. We've got a little bit of a login page. In this case, it's a very simple one where annotators can log in with a password and that's stored. These Anki flashcards, none of this is sensitive. They're the, you know, the data for their annotation set is in the GitHub repo. They've kindly allowed that to be public. And yeah, you can also set up OAuth or anything else that you need. The main page is a dashboard, and this allows any annotator or really anybody that goes into this app to see like, okay, here's what query we're talking about. As you can see, some of them are PDFs that they can link to, some of them are text queries, and there's others that aren't on the screen we can look at later that are just very one-word queries. Each annotator— this was kind of built progressively— each annotator can see what's the progress on that. And so you'll see down here the idea is that they want 2 annotators for each one, and they want them to be complete. It's a multi-step process, so 2 annotators should review, You can see that this first one we had a little bit of confusion with more annotators annotating it, some of them not finishing it, that kind of thing. And that led us to create the status screen. And I think that's a big part of these annotation tools is get something quick and easy out there and fix the things that cause confusion rather than trying to spend a week creating or 3 weeks creating a really perfect solution. And so we added these. Once two annotators do them, it goes into a finalize step. And where that is, is we don't need another step if both annotators agreed. If the annotators flagged and rated things differently, then they each need to go in and do a final rating and put in a note for why— and put in a note for why they feel— what their final rating is. It might be like, oh, I misread that, it really should be, or I misclicked that, or I don't know why. Or it could be a more detailed technical explanation like, this is why I don't think this is relevant, this is why I do think it should be reserved. And then you can, you know, figure out what the tiebreakers are. These annotators are actual medical students, so they are actually the domain, and they do work constantly with the product manager. So again, that kind of gets back into what they were saying is like you need domain experts to look at this. I would love to go in and document medical flashcards. However, a product of what a software engineer or an AI engineer thinks is good medical results is not what you want. You want those domain experts. And so this allows them to do that. This evaluation page, each query, which you might notice here is you have a fairly short query and there's 400 documents, relevant results that are coming back because they're all very, very short documents. They're all flashcards. And so for this case, these, and this is a nice addition from LabCodes, is All these keyboard navigations really reduce the friction and make it easy to go through 400 or more of these results for a query. If you're doing an annotation where, you know, reviewing a, a very long conversation, each one takes 2 or 3 minutes, the keyboard shortcuts might be less important. If you want to have, you know, 400 very tiny results and you want to go through them quickly, it starts to starts to make a big difference. We found that there was only two things we want feedback on. First was, is this relevant or not? And of course there's a tiebreak— that tiebreak system later. But then second, is the query actually good? So for example, if someone types in heart and that's their query, do we really want to, uh, return a card for everything in the medical domain that involves the heart in some way, shape, or form? That's a lot of cards. And so they can also give feedback on the query, like, this is too general, or I don't, I don't know how to interpret this. Should I be, should I really be rating this for, you know, potentially returning 6,000 flashcards? They can't study all those at once anyway. And that's, maybe we need to be able to have a system for asking for user intent or follow-up system. And so in this case, we're rating not just the, the results, but also the query. And these queries came from actual production traces that users entered. So we know these are actual queries that they're interested in because that's what they searched for. We have the simple good, bad, clear. This is nice, just having them lined up. You know, it looks kind of janky because it's like, you know, it'd be nice to have you know, a prettier thing. But what we found is like just labeling it this way allows them to scan 400 results very quickly and see what they are. Is it good, bad, or clear? And so kind of prioritizing that functionality. This is what that finalized step looks like. And so you can see here when they go in, it's like, okay, there's, there's 4 of these that aren't aren't rated yet, that the annotators disagreed. They can see what the other people put, put their feedback, and put a final rating. If they agree at the end, then, you know, we'll be reviewing it. And so these comments here are what goes there.

So it should be noted that like, okay, already we could see In this application, there's like a lot of applications or domain-specific stuff, like, you know, annotating the specific query. In this case, I happen to know some inside baseball. And so here, you know, we're trying— they're still working on the rubric for the annotation. So they're not just measuring the annotator agreement. They actually want to have a discussion if there is a disagreement, because they're trying to— at this point in their journey, they're trying to work on that rubric. So, so that's why there's this workflow. Whereas, you know, if you take like a most annotation tools out of the box, like, yeah, they'll just be like, okay, well, here it, here it is, like, deal with the disagreement. And we'll talk about that more in later in the book and also in the course. But you can also already start to see like, this is why we advocate for building your own because there's a lot of domain-specific stuff. In, you know, product-specific stuff they might want to bake into the annotation tool specifically.

Yeah, absolutely. Thank you. The last piece of this app before we start talking about the code to build our own and actual FastHTML stuff here is this push to Phoenix. Phoenix is somewhere where you can store traces annotations, datasets, experiments, prompts, versioning. It's a lot of stuff. One way to think of it is it's kind of like a database for your AI experimentation. I don't know, maybe Hamel can explain it better, but that's what we're using it for in this case is for each of these queries, once it's done, we can, you know, push them to Phoenix, we can delete them from the current thing, we can bring things in. And this really makes it very easy to adapt, adjust, and annotate new queries without, you know, constantly trying to add it in one place and then upload it into a database in another and go back and forth. And so building a little bit of this simple admin dashboard was helpful.

So there's a— yeah, just a little note about Phoenix. The guest lecture right after this one So at 12 PM Pacific time will be about Phoenix from the lead developer of Phoenix. There's a lot of evaluation tools out there. The market is exploding with different tools. You know, those tools allow you to have observability, kind of like a database, like Isaac said. They also offer prompt playgrounds. Yesterday, last time we saw Braintrust, for example. Phoenix is open source, so you can, you know, play with it freely. That's why we have it in this course, and we'll be talking more about that, and Isaac will be kind of showing that as well, hands-on.

Oh yeah, definitely check out that next lecture. It's going to be very relevant. Okay, so that's what was built. That's the actual use case. It's an actual commercial product. It's actually making money. So that's the premise for this talk. What is FastHTML? I've got a couple of slides on this and we'll move on quickly. Basically, it's a web development framework for Python, and the goal is minimal boilerplate, very, very simple developer productivity, and very little language switching. You know, when you make web apps, you're often switching between Python, CSS, HTML, JavaScript, or some combination. You don't avoid that completely. The web is an HTML, JavaScript thing, but this is designed to really keep you within Python. And for this application, there's very little or no, you know, custom CSS or anything in there at all. So should you learn it? Tool's not what's important. FastAPI, Streamlit, Shiny, they're all good options. I like FastHTML. It keeps me in Python. I can work very quickly with it. If you want to be able to build web apps like this super quickly and you can't and you don't know where to start, I would say start here. If you can already do that, you know, no need, but maybe you'll find it interesting. But yeah, it's not like you can't build an annotation app in others. And actually, in the repo, there is a FastAPI version as well you can check out because There's lots of tools you can do this in. FastHTML is just the one I like. All right, this is the most minimal version of a FastHTML app. This is a fully running app. You can put this in a Python file, hit python main.py or whatever you named this file, and it will run in the browser. You can go to localhost:5001. You can deploy this to Railway. There's no extra config files. So that's, that's kind of the, the theme here. Is that there's no HTML files, there's no CSS files. This is using FastHTML with Monster UI. I'm not going to cover a lot of FastHTML in a ton of detail. There's more resources in the GitHub repo. My goal is kind of give you a little bit of an overview of what's possible. This is the same thing. Sorry about that. All right. For this example, we're going to be using a database. This is using a database connection thing called Fastlite. Not the, the point of today's talk, but you can see there is a single table defined in Python where you have IDs for both your input, what the user input, and the document that's being returned. And you have a field for the query content, the document content, annotator notes, and whether it's good or bad. This is a simplified version of the, the commercial one so we can cover it. We create a database and we create a table. So Fastlite is another tool created by Answer AI that kind of does the same thing FastHTML does for databases, kind of lets you do a lot in Python. This is the main page of the simplified app. You can see it's, it's, it's kind of similar. We don't have the finalize step. We don't have all the all the tracking of which annotator is doing what and at what stage, but we have the query and we have a button to evaluate. So that's the core kind of MVP we're doing today. This is the code for that route. There's a few things to keep in mind with FastHTML. This @rt, this defines a route. If you've used FastAPI, you know, or other things, you might often have a decorator that has a string that says like this route is the slash string. FastHTML does that kind of automatically for you by default. You can still define it if you'd like, and it does that by looking at the function name. Function— a route named index knows is the root route. If you have a route named hello, it will be /hello, etc. And so this creates this route and we get all of our unique inputs. This is using going into our annotations table. We're just, we just have the one table and it's grabbing the distinct input ID and input. So this will give us all of our distinct queries. So the things the user actually enters for the search, we go through these. And we're going to create a dictionary. The dictionary has our user query and a link. A link, um, we styled it as a button. That's all this is doing is saying make it look like a button instead of a link, but kind of same thing. This is a, a link to the evaluate route, and we can see this link goes to this evaluate. So there's a few things you might notice. Is that this href is actually just linking to another Python function. We haven't seen it yet. We'll show that in the next slide. And it's passing parameters to that particular route. So in this case, it says, okay, if I click on this evaluate button, I'm going to go to the evaluate Python function slash route. Thing and pass it the input ID that it clicked on. Now there's a function called table_from_dix that builds this. So one key thing I wanted to talk about here is there's a few things you can notice here. You, you can see this route is very Pythonic, lots of stuff in Python. You don't see a whole lot of HTML. However, you are writing a lot closer to HTML than, than you might think at first. So when we look at this table_from_dix, this feels a lot like large abstracted functions that don't give you a lot of flexibility. You know, however, when we look at what the tableFromDix function does in the source code, all of these are HTML elements. So for example, we can say like, oh, what is the, what is the tr element here? You can actually go to the HTML spec and it's here. You can see there's tables, there's table rows, there's table heads. These are using them directly. So we're not doing big transformation. A button in FastHTML is in an HTML sense, a button. The transformation is we're taking away the pointy brackets and we're using parentheses to match Python, rather than like a big, like abstracted different thing. So that's one thing to keep in mind. That means you get a lot of flexibility, more flexibility than, than like a, a Gradio or, or a Streamlit. Yeah, so that's there. So when you click this evaluate button, it calls this evaluate route. That sends you to this page. This evaluate page shows the user query at the top. So in this case, this is the user typed in coenzyme Q. And it shows a list of all the different things that might be relevant to the query. And the annotators need to mark whether it's good or bad. And if there's a comment, that comment is. Just for this one coenzyme Q query, there's 400 possible results. And they could put whatever they want. They can say, you know, here, this Yvonne Willibrand is about clotting, whereas coenzyme Q is about cell energy. They can say, I think, no, these comments are made-up comments. These aren't the real comments. These are written by me, so they're not great comments. But the idea is they need to go through and make a selection and make comments on the ones that are relevant to 400 just for this one particular user query. This is what that page looks like. This is the evaluation page code. You can see the same pattern. There's a route that creates the route, and we have the evaluate function. So we know that this route is going to be at /evaluate. So if you go to myannotationapp/evaluate, that's going to be this page. It just pulls it from the Python function name. And there's one parameter it takes, the input ID. If you're familiar with web apps in general, what this is doing is it's actually going into the request object and it's automatically, it's looking for an input ID in the request that would be typecast and make sense to typecast to a string, in which case that's most everything, and it's automatically pulling that out. You could put in a request here and use that and pull it out yourself. This is just a convenience thing that goes into the Starlette request for you so you don't have to do that yourself. In this case, instead of listing all the unique queries, We're grabbing all the rows for a particular input ID. We go through every single one of those documents, and again, we're going to use this table from dictionaries. And so we do this, so we say, okay, we're going to grab that document, and first we want it to look good, so we're going to render it with Markdown. There's a component in, um, Monster UI, that's render.md, uses Mistletoe behind the scenes to convert to HTML, and then it just inserts Tailwind classes into that. So it styles it with the usual HTML approach to styling. And we have this notes. So let's look, output notes and evaluation. So the output is the actual content of the card being retrieved. The notes is this text field where they can put in comments, and the evaluation is a good and a bad button, or relevant and not relevant, but relevant and not relevant really didn't fit on the page very well, so good and bad. This input will grab the note from the database if it's there. If it's not there, it'll be none, it'll just be empty. You know, we used— I used Tailwind here to change the style a little bit. I felt like the input was a little bit too small, so it was hard to type in. We can just go ahead and use Tailwind when the library doesn't do what we want. Um, and we make a POST request to a route. We'll talk about that in a second. Uh, and this is where we start talking about what HTMX is a little bit. HTMX is, um, um, a paradigm where this POST request is going to call a route. In this case, we're defining it as a Python function, but it's going to call a route And that route is gonna return HTML. But instead of returning an entire page, it's just gonna return a small piece of the page. And that small piece of the page can be put wherever we want. By default, it's gonna replace itself. That's the default, so that's what we're gonna do here. So it's gonna go to this update notes route. It's gonna pass in a couple of arguments, the input ID and the document. So what is this particular document? It's going to get whatever is returned, in this case HTML, and then it's going to replace itself because that's the default. And we could tell it to replace something else, in this case replace, replace itself if there's something back. I think in this case we don't return anything because we don't need it to change. And then we can define, well, when do we want this POST request to go? When do we want to request that information? And in this case, we want to request it anytime we change it. If they type in stuff to the— it's not how they type, but they type into stuff to the, to the input box, we want that stored in our database. We don't necessarily need them to type and then hit a button. So every time they change, it's going to update that. And then the last thing we have is we have our evaluation buttons. This is another nice thing I like about FastHTML is that All of this UI stuff can be refactored with Python, so I can use normal Python abstractions. I don't need to learn a templating language, I don't need to do any of that. I can just keep using Python for however I want to abstract it. Lots of ways to do that, but, but you can just use it the regular way. This isn't a route, this is just calling a Python function which is going to create that component. We'll look at that in a minute as well. And what you'll see here is we we render our document input. We can process it with some Python to make it look, you know, get rid of some, some styling we didn't want. We can, um, uh, I will get to these questions in a minute. And then we render this table. And so that creates this route here. Okay, so we looked at this and we said we're going to call this update notes, which is going to every time we trigger a change. Let's see what this does. This update notes route is very small. It takes in all the arguments we passed, which is the input ID and the document ID and the notes, and, um, it's going to create this annotation record. If we go all the way back to where we defined our database— sorry for this— we see we actually created a class, and that class defined the schema. Well, this class can be reused to actually update our, our database. And so we create this annotation object and we'd say, you know, update the database with this record. Because we're not returning anything in this update notes route, there's nothing to update on the front end. So in this case, that's all it's doing. Every time we type in a change and it changes, it calls this route which updates the database with our new changes. The evaluate buttons are the one of the last things we're going to talk about here. We can create IDs, and this is getting a little bit trickier, but as you can see, we're using a button. We have that same pattern. We're going to do a POST request where we're going to evaluate it, evaluate the document, which is going to update the database. In this case, we're doing this h-text target. This target is saying, okay, I'm gonna call that route. Instead of replacing myself, I wanna replace something different. And so I'm telling it where to target. The default is to replace itself, in this case, the buttons. Um, uh, in this case, I'm gonna tell it what to target. The reason I'm doing that is I wanna replace both buttons. And what we'll see here is we're going to make this button a primary color, color that really sticks out, you know, if it matches the label, or a secondary if not. And so we're going to have a good button and a bad button. And so if in the database we've stored good, it's going to be this blue color, it's primary color. And if we've said bad, it's going to be that gray color. And we see that here. If it's, if this record has bad stored for the evaluation, the bad is the primary color. If this annotation here is marked as good in the database, then blue shows up, just giving a little bit of a visual thing. And so what this does is once I hit this good or bad, it's going to replace just the buttons, not the whole page, just the two buttons, the container that has those two buttons. And update the colors. And as we see here, when we evaluate the doc, we create our little annotation object to update it. We update the database and we return the evaluation buttons. And these are the new ones after the database update has happened. And so we get that color change. Okay, so that's the app. I know I covered that super quickly. I don't expect everyone to like, have understood all of that, but I hope, I hope it helped understand a little bit of, of how things work and where you might start and what you might be able to do with it. If you want to deploy stuff, Railway is great. I do that a lot. Um, Plash, Answer AI has a hosting service. It's currently in beta, so if you're looking for like the most stable thing, maybe wait until it's released, but that's available. Um, any Starlit-compatible hosting. Um, in the repo, there is a link to where you can find other deployment options, including Modal. So if you've got modal credits, that might be a good place. And yeah, here's the companion repo with all the stuff I talked about. I mentioned Wade helped a lot with the FastAPI React version of the app that's there. Here's his website. So yeah, so definitely check him out as well. That was very helpful. So I think—

Really appreciate that.

That was great. Um, so I guess I'm gonna just try going through, um, and ask— answering questions in the Discord. If you have a burning question that wasn't answered, um, feel free to re-ask, but I will do my best to answer them, uh, unless anybody— or Hamel Husain and Treya, if you've got specific questions that you saw on Discord that you think would be good, feel free to jump in as well.

Yeah, I had some questions that I always ask you privately, but let's just— because they're probably interesting, so I'm just gonna surface them publicly. Like, okay, so is it a good idea to be developing things with FastHTML? Like, you know, AI seems like it's really good with kind of the mainstream stuff like Next.js, so on and so forth. Like, are you swimming upstream? If you're trying to use FastHTML? What's your perspective on that?

Um, yeah, so, um, there's kind of two kind of extremes that I think of. There's like, you can code completely without AI, and I don't think that's a very good idea. Um, AI is very helpful. The other extreme is you can like vibe code without looking at the code at all and just rely 100% on AI. I don't think that's very good either. Um, I do vibe coding for like a lot of brainstorming and stuff, but I find even when I'm using, you know, um, vibe coding with like Next.js and React, I find I often get to a point where I find some bug and I'm having trouble getting the AI to fix it, and then I end up needing to dive in, and then there's tools and frameworks I don't really understand, and it's just, um, I don't know, maybe I could dive in, but it's painful experience. Personally. I know people that are really successful doing this, but I find it best to hit a little bit of a middle ground where I'm heavily using AI, but I'm also doing like a lot of refactoring and knowing roughly what's going on as I go. So with that in mind, this entire app is about 80 lines of code or 70 lines of code that I looked at, And so that's really nice for, from a can I understand the code perspective. Of course, AI doesn't know FastHTML as much as a more popular framework or an older established framework. And so it does, it is a little bit harder for that. Now, one thing that I have in the repo that I found helpful is I use, oops, this is not that repo. I use a lot of these cursor rules to, to help with this. And so you can see here, this is a cursor rule for FastHTML where it goes in and it gives lots of examples that can be used of how to use it. I find that if you give the right context to the model, I mean, same thing as with anything really. If you give the model the right context, it does a lot better. You know, than if you just don't say anything and do that. It already has a lot of context on Next.js. That means there's a little bit more work going into, um, into doing it. I've explored a lot with the difference between like FastHTML and using like FastAPI with a Next.js frontend and then using Next.js. Um, I find the Next.js stuff to be really appealing. I'd really like to use it, but so far I haven't found a way that I enjoy it, and it always ends up being more complicated than I like, but makes sense.

Yeah, I mean, I gathered from talking to you, like, okay, you know, the sweet spot is like, use what you know, and then if you're using something slightly outside the mainstream, if you supply good enough cursor rules, then it seems to work. And that's, that's my experience too, like cursor rules really help a lot. What's your, what's your opinion between like, okay, so there's this like FastHTML, this is like basically Python like completely, and then there's FastAPI, which is like Python on the backend and whatever you want elsewhere. Like, you can like mix. What's your— what's the trade-off there that you like?

Yeah, I, I really like, like them both. So like, what I like about FastHTML is super Pythonic, it's short, um, I, I can do everything in Python. Um, but I do like— one thing I really like about like, uh, and Wade helped with this FastAPI version One thing I quite really like about the FastAPI stuff is it's really, really, really nice to have a full JSON, like, API that you can call and you get dictionaries back. You can use those in notebooks very, very naturally. You can, um, um, test that backend and have the frontend decoupled. Like, you can do that in FastHTML. It's not like there's an hard limitation, but, um, creating that JSON API as, for that core functionality and having that decoupled from the front end, there's, there's a lot of really, really nice things about that. So that doesn't really answer the question, but I mean, basically I like both.

How long does this, how long did this like annotation app, like if you're going to create it, what was the level of effort essentially of this that you showed, the Anki one?

Yeah, so the Anki one, I think came together over the course of, uh, I think it was like start to finish it was like 3 days, but they weren't like full days. So I don't know, um, not a ton of time. Maybe I, I probably could have sat down and done it in 5 straight dedicated hours including all the finalized steps and the, you know, the logic of the, of the database and the the tie-breaking and all that. Um, so it comes together fairly quick, but I think it's, um, it's also important to say, like, I wouldn't necessarily— I wouldn't recommend sitting down and trying to define too much of the finalized flow and, like, what the buttons should look like exactly and, like, where all the fields should be and how the stuff should be rendered and what keyboard shortcuts you could— should have. Like, I wouldn't try and define that and then just like sit down and do the whole thing, because you're probably going to build a bunch of stuff wrong anyway. Um, when the annotators start annotating, they're— you're going to find stuff. Like, we didn't build— we didn't— we hadn't even considered that we should basically do open coding on the queries before. Like, we didn't consider that like we should be evaluating if the query even makes sense. And there's going to always be something like that that seems so obvious in hindsight that you should have it, but you just won't think of until you actually look at the data, which is why you have to look at it a lot. So yeah, get something simple that's fast rather than defining a lot up front, in my opinion.

Sounds good. You know, We can take questions, but we can also give people a break. Um, there is the next session's at 12, so why don't we retire, uh, give everyone a bit, you know, almost 15-minute break. Uh, and there's another guest session on Phoenix, which we'll be using in this course.

So, uh, feel free to keep asking on Discord. I'll, I'll, I'll go— I haven't read any of them yet, but I will go through and read them and answer into them. So feel free to keep asking there.

All right, sounds good. Thank you for coming on, Isaac.

Yeah, thank you. This was fun. Bye.

See ya.