Paired Error Analysis With AI Agents

Today we're here with Isaac. I've been working with Isaac for a really long time, both in open source and across some companies. Today we're going to be live coding an annotation app using something that he has built, but I don't want to ruin it. I'm gonna let him tell you about it.

So a few weeks ago, or I guess it's been almost 6 weeks ago, I started working with a team at Spec Story, and we're getting close to a beta launch. So we're going to show a tool that we've been building. You can kind of think about it as taking Claude code capabilities and putting it inside like a Zoom app with Google Docs kind of all together. Okay, so this is Stoa. A couple of things that you'll notice, we are actually in a live call. You don't have to be in a call to use Stoa. You can see we have files here. You know, I can create a Markdown file and Hamel can go into this. This is Google Docs-like. So as I type in this, Hamel can see it, he can edit it as well. I can see his cursor as you can see. And so it's got this like live Google Docs collaboration thing, which we'll look at and we'll look at building prototypes. and we have our agent aspect here as well. And so these are kind of the components that we'll be using. What we're going to do today—

so far, is it like Zoom meets an IDE meets a coding agent? Somehow you mash them together and you get superpowers?

Okay, that's right. And it's Google Docs mashed in there too, because, you know, the live Markdown stuff, right? And this agent isn't my agent. It is my agent, but Hamel could start his own. So he has his agent. Because we're in a call, we can see each other's agents. He could ask my agent to do something, and so It really is like a multiplayer agent in a sense. These are kind of all the pieces. What are we gonna do today? This is an evals thing, so we're gonna use it for evals. You can imagine this is a pretty general thing. I got this compliance form, which is a little plugin you can customize to add stuff to your app. You can see it's got the official Parlanx Labs logo there. I'm pretty sure that's what Hamel uses all the time. And the main thing is looking at your data. You can see it, it's kind of repeated many times. And so that's what we're gonna try and do. We're going to build stuff to look at data. And one of the nice things about this environment, in my opinion at least, is that because everything is collaborative and you can see the agent, I can build this with somebody who doesn't code, who isn't good at using AI. Maybe they're a lawyer, maybe they're a doctor. And rather than me building the annotation app and handing it off to the domain expert and then kind of them giving feedback or, you know, screen sharing, we kind of build it together. Hamel Husain talking to my agent here. So what does it look like to look at data? So there's a few ways to do it. I'll show you the first two and then the third Hamel and I will build collaboratively. We're just gonna tell it so we don't have to wait. Start a server to serve index.html. So the first is this data viewer. This is a raw data file. This is a JSON-L file. It's often what you might see from like an Agentic application. So if you have a product that's doing JSON-L, this is often the storage mechanism. Claude Code. And Codex, for example, store it. So if, like, if you go into your home directory,.clod directory, behind the scenes, all of your Claude code history and, like, your tool calls and its thinking steps, all its prompts and its plans, all of that is stored in a big JSONL file. As you can see, this is horrible to look at, but often it's very fast for a development team or for people to export into this format faster than doing it using this plugin. And so this makes it a little bit easier to view it. So this sits on top of this particular type of file and allows you to view it in a little bit nicer way. And so you got filters and stuff. And so the nice thing here is that now you can build this and have some way to look at this question answering dataset. The second thing that you can do is you can have an open server. And so in this case, this is a server and I'm viewing just a different dataset. In this case, it's a tool calling dataset. I just grabbed 3 datasets that I found.

How did you click on that link? Because like, I think it said localhost. What's going on? You clicked this localhost and it opened— it created a tunnel for you to do something?

Yeah, well, I opened it up here. I'm going to go ahead and share this with you. Over in this left sidebar here, there's a running app. Now it is running on localhost:8080, but this Claude Code type thing is actually running on a sandbox environment. And so you can click around and you can see the data. I didn't put a lot of effort into this. I just asked the agent to write it. So these are two ways you can make more comfortable viewers. The third thing is, um, I thought we'd do is we'd actually make an annotation app and not just a viewer. Looking at the data is cool, is convenient. You can do it based on this JSON file. This is actually what we were just looking at is this big JSON file of stuff. And so it's really nice because if you have any kind of application, you can usually very quickly export as a CSV or a JSON or JSONL file and build a viewer very quickly with an annotation. Like I said, these are— I have to go through this history, which all of the chat histories are stored here. But pretty much all of these were like 1 or 2 prompts to create. And so what I thought I'd do now is Hamel Husain and I could create— there's a third dataset, and I thought we could start from scratch and see where it goes. I don't think Hamel Husain's seen this dataset before. I just kind of picked it out. I haven't looked at it super carefully, but I thought we would look at the dataset, understand a bit, make a simple app, try and do some open coding, try and edit the app based on what we see. Yeah, just kind of try and do the process on this particular dataset. How does that sound?

Yeah, let's do it.

So the first thing, it's like you can look through and see what it is. There's a dialog ID and services and turns and utterances. And all this stuff. The first thing I might do is I'd look briefly at the JSON file and I'd see, yeah, okay, it looks like some sort of chatbot. We already have the server running and I'd say like add a second data viewer, call it multidialogues.html to look at our MultiWaz dataset. I went to Hugging Face and I put in AI datasets and I downloaded the first 3. I didn't wanna like cherry pick the best ones. I just picked 3 that looked different. I don't really know, to be honest.

So it looks like location-based AI stuff. There's someone calling a taxi. Let's see someone else trying to see what restaurants they should go to. Yeah. It's a lot of location-based stuff. Like, like you might do as a tourist who are visiting somewhere. Like I'm looking for a nice place to stay and then like some kind of travel buddy. Yeah.

Okay. So we've got the start of an app. We're having genuine AI stuff. It has an issue. And I'll just tell it, I can't see the actual examples, even though I can see the general shape. Here's the error. Do you need to restart the server? And I think where this becomes most useful, well, this is kind of a simulated thing because neither of us know this dataset super well, but it does tag intents and things like this inside these frames. You can see it tags that there's like a restaurant and it goes deep. But where this would be helpful is if this was a dataset where You know, maybe it was a legal application and I was on a call with an actual attorney and they were telling me, you know, this start isn't very important. Oh, this slot, this is actually really important for legal outcomes. This is like the most important thing that we need to look for first. And those are kinds of stuff that it's really hard as a developer.

Yeah, I'm going to tell the agent to do something too. Each conversation in this dataset looks like it has services. Can you make some kind of bar chart with all the services on the y-axis and the x-axis have the count of each of those services that are invoked. Go ahead and put that visualization in the same HTML application as a separate tab or something like that. Okay. So yeah, you can see what I'm saying.

Yeah. So now we can see, we can search, we can filter. I can see I'm looking for an expensive restaurant. You have to expand to collapse down, but that's how annotation apps start. So you got to do it and it's like, Oh, restaurant, food, Caribbean. I think it identified that properly. And that's a simple case. And this is interesting. It's saying the price range is expensive, but, you know, it's not surfaced to the user. It's tagging this should be an expensive restaurant, I guess, from their initial turn. And so these would be kind of some of the states in the center. Okay. So it's kind of tagging some structured data based on the conversation, I think, to be able to find stuff. And so these would be things I don't like that it's hidden because this feels like You know, one of the big things, this is like tracking the user intent to find stuff.

Is that the stu— a tool called Result? This like collapsible thing. Is that what that is, you think?

I think that's these frames here. In this case, I think it's tagging and it's identifying what is the intent of the user. What is the user trying to do? In this case, it's saying like, hey, their active intent is to find a restaurant. I guess it's almost like a planning feature. Like what is the user trying to accomplish? Their active intent is to find a restaurant. And they've got a couple values here. They're looking for something near the center of town that's Chinese food. In this case, like taxi, and there's no active intent. So that's kind of interesting. And so it's saying they're looking for a restaurant. They're not looking for a taxi.

Okay. I'm going to tell the agent to do another thing. So it looks like the dataset has an active state in each conversation.

So in frames.state, you can see as he's typing here, I can see what he's typing before he submitted it. That kind of helps prevent the multi-agent stepping on each other.

Which shows the current user's intent. I want you to also visualize this in the stats tab somehow. Maybe visualize it as a graph and we can count how people normally flow from one state to the next. And you can label each of the edges with a count and also make the width of the edge correlate to the count just so we know how people tend to flow throughout the state. Also probably want to add some beginning state and end state to make the visualization comprehensible.

And this is that prior chart that, that Hamel created here. Pretty cool. Probably the first thing I would do is I'd see like, oh, train and restaurants is like most of them. You might be able to kind of look through or you might be able to create a new thing like a filter for trains. And so I can filter for trains and see how many are there. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Yeah. It seems about about right. Taxis are, you know, you can kind of guess this. Okay, 3, 6, 7. And so that's not a definitive thing, but it's like, oh, it happens to match. And then you might want to dive in deeper. Oh, it's using D3. That is a library I've tried to do manually and it's painful. So I'm glad the agent's doing that.

I would never do D3 manually. It would take hours, maybe days. I'm serious. Cause it's like pretty low level.

Do you want to start a second agent so we can do more stuff as well?

Okay. I'm sorry, Mauno.

I've got a, uh, try here. You're agent and advise what I should do next based on this person's writing. While it's working on this, there's other stuff that's useful. You know, it does do a transcript of the meeting and the agent can see the current transcript as of up till now. The nice thing about that is then Hamel Husain and I could talk and critique some sort of view and debate it. And then we go back and be like, hey, look at what we just talked about and fix it based on what we decided. We don't have to necessarily type it again if we're clear about our discussion. So you can see here. All right. We, I don't know. I don't know how useful this is.

Yeah, it's a little bit. Yeah, yeah. We have to clean up this thing.

Yeah. Look at how much data we got. This is definitely the wrong way to look at data. Anytime you—

It's the wrong way. Yeah. I would probably get rid of the start and end nodes for sure. I can see that fine train to book train definitely seems like a common path.

Let's get rid of the node. Too confusing. Let's do a simpler table-based view. Draw inspiration from transition matrices or something. I don't know exactly what it's gonna do there, but we could give it something else.

Maybe a sanity diagram would be fine.

So what are we doing in the multidialogues.xml? Advise what I should do next based on Hamel Husain's evals FAQ. Let's see how good this is. So we got a Data Explorer.

You can use the skills. What about the skills? You want to try the skills?

I don't think we can load in skills, but I guess it has the repo. It should be able to look at the actual skill files.

Yeah, it's just a prompt. Let me see. Let me give you the skills. Let me give you the repo.

What I should do next. Run your LLM on a sample and load those into your viewer. Great. Add annotation controls. You need to manually label each example pass/fail and note why it failed. Seems like good advice. The browser is missing a judgment column, which is a pass/fail, a failure category tag, and an export button for labeled results. And number 3, don't automate yet. Don't build our LLM as a judge or automate metrics until you've manually reviewed enough examples to know what failure modes actually exist in your specific application. The intent transition flowchart is interesting, but it's not Eval's work. Okay. I didn't like our chart either, which is good.

Okay. Yeah, this is way better. Start.

This is nerd.

You find a train, you book a train, find attraction, do a lot of stuff. This is worth lingering on this for a moment. This is not just eye candy, like, oh, I can put some numbers on the page and whatever. First of all, I would want to understand why is there not book taxi? Is something wrong with the data? Potentially, maybe there's not. Maybe this is the way this application is. You don't book taxi through this. I'll be like, hmm, what's wrong here? Why is it not book taxi? I would look to see if there's anything fishy here. That's the main deal. So he's like, okay, it looks like people are from the start. They're kind of equally finding trains and finding attractions and finding hotels. The finding taxi thing is interesting. Like, people don't want taxis or what? Like, why is it so low?

It's weird that there's more trains than taxis.

It is weird. Yeah, I would look into these things. Why is there so few taxi thing? Is that a problem with our app? Could be a problem with the AI, could be a problem with the app, could be a problem with anything. I don't know. But like, it is interesting. I would want to do an analysis on this find restaurant. Things. Like if you find restaurant, it would be good if they book the restaurant. You would do another visualization that says like, okay, for find restaurants, what ends up happening?

This is find restaurant. I think this 7 goes to booking restaurant, right? But then almost the same number after finding a restaurant go find a hotel, which seems odd. And it seems odd that there's more trains than taxis. You can find an attraction and a taxi, but both of those you can't book.

When you explore the data, you come up with some hypotheses like, oh, okay, this is weird, this smells funny, etc. It can feed into like synthetic data generation when you come up with your dimensions and you like try to stress test your application or you want to dig into some areas like this is how the whole process of data analysis works. You want to first take some time and understand the data.

Yeah, so I guess what I'll do in your agent here is this thing based on your writing said we should add annotation controls. Maybe we're not ready for pass/fail. Add a notes field so we can take notes on each one. Have it save data in like a JSON file to store so we can see what's already noted and pick up where we left off. Was there a visualization that you wanted to do, Hamel Husain?

Yeah, let me go to the agent and narrate it. I want to add another visualization that is anchored on finding a hotel. And I want a Sankey diagram that shows the steps in the flow of everyone who starts with finding a hotel, where do they end up at?

Actually, all these files are CRDT-backed, which means that basically there's a version control all the way down to the character level. So yeah, all of these are versioned. It's not like Git where it's, you push at a specific time. It's kind of live streamed through and all that history is saved. So actually what this agent is doing is it has synced this project onto basically a regular different machine. And it has a— it uses our— the Stoa CLI. There's a little CLI here. And so you can clone it to your own local machine and use your own local Claude code or codec or whatever your tool of choice is. You're not limited to using this agent, but it syncs that there and it's a two-way sync. And so as this edits, it's just syncing back into our app here. But again, just like the agent, you can clone or you can, you know, do remote sets and all that. Just like the agent is just writing to a different machine as two-way sync, you can two-way sync to your own laptop or anywhere you want. And all the history is there as well. I would say the version control, it's more like the Google Docs than it is more like GitHub where you do like a huge chunk and then that gets committed in intentional places. Alright. So now we've got here, we've got this notes field in the browser. We could write notes. Let me just take a look. I can actually look at the data. So right here, looking for expensive restaurant, center, expensive. Great. It's looking for Caribbean food. It kept tracking that. It's still tracking. So it's asking for 2 expensive European restaurants and 1 expensive modern European restaurant. That seems like a really weird question to me.

I'm looking for an expensive restaurant in the center. Who the hell says expensive? Okay. Anyway, I can recommend several, uh, no matching restaurants. What about restaurant that serves European food? I found 2 expensive.

So what's the difference between an expensive European restaurant and expensive modern European? I guess these 2 are old. Not modern. It's kind of a weird wording, I guess.

Yeah, see, at this point, so if I was doing error analysis on this, I would say, look, you're doing way too much in text. When it finds the restaurants, at this point I just want to see a widget. Let me choose or show me the 3 buttons. I'll be like, I'll click one, you know, because I can then like go look at the websites or get a sense if I wanted to.

I think it's also like, I'd like to book at this time. It would be nice to have a picker so I could see what's available, especially if that time isn't available.

Wow, just for yourself. Wow, that's surprising. I thought this would be like some business thing? Because why they want expensive?

I need to go somewhere expensive by myself. It must have been a rough day, I guess. Like, this is really bad. Your reference number is here. Okay, from where? Where do I go to use this reference number? Like, that's what company is to train with?

Yeah, that's really bad. What are you gonna do with that? You should get a link to something. Yeah, at this point, like, you would have failed this. We could have said, like, hey, this needs a widget. That wouldn't be something you need an eval for. Because that's just the product problem. Like, hey, this product is not working the way you want. We need to redesign the product. This is like very frustrating. That's part of error analysis is like, okay, you look at the data, you actually see what is wrong.

Cool. So that's open coding right there, right? I need a train from Birmingham New Street to Cambridge. There are 133 trains traveling on that path. Uh, okay. I mean, I guess that's L for the dollars a lot. It seems kind of weird.

This is weird again. This is bad. Again, you need to show a widget, like some Google Maps widget. It says like from destination to destination. The person can change the address or do some things like, do you— is this is where you want to go to and from? And like, what the hell is TR1670? Do you go around knowing the names of these like trains?

No, I'm like, I don't know what company it is. I don't know if it's business class or economy. I don't know how expensive it is. I guess $20 may be great. I don't know how long the train ride is. I don't know if it's a sleeper car. Yeah, I don't know. They follow up with how long, and it's 163 minutes. Like, they have to keep asking detailed questions.

It's like there's not any product design going on here. Like, someone just lazily said, oh, let me just put a chat app on tools. Let's have a chat app without any more further thought.

I don't really like this. It's like, what is the exact arrival time? I'm also looking to place to dine in the center of town. Expensive modern European restaurant again. And it's like, let's take care of the train first. Do I have to? Can I figure the whole plan out? Like, do I have to book the train first? I guess they have to follow up. Is there a particular type of cuisine? I would like modern European food, which is kind of bad because they just said the modern European food. So it did track the state, still has a state here, and yet it asked again. So that's interesting. Like, did it not send it? I don't know what you think, Hamel Husain, but like, I'd probably look at 2 or 3 more of these, but I'm pretty sure that like a priority pretty quick would be this like visibility and widgets and like figuring out how to make these IDs useful before most other things.

And that happens a lot of times in error analysis. So like, well, okay, time to stop. This product doesn't make sense. It's usually not this bad, but you know, there's a spectrum.

Whoa, what is this?

Yeah, something's wrong with the send keys. Book hotel shows up fair amount of time. It looks like there's some churn. There's some places where you're like finding hotel roughly, I guess, like half of that time you find a hotel, you don't ever book a hotel.

Do explain this diagram to me a little bit? Sorry.

Yeah, it's basically like, what does your journey look like? Like basically out of all the people that find a hotel, what does their journey look like? Half of them book a hotel, half of them go to find train. There needs to be some deduping. There's like some problem with this diagram. Probably they need to debug it. Let's just pretend that it's the right visualization. Let's say this travel application is like, you're trying to monetize it and you can get a commission from hotels that when you book the hotel. If you find any, trying to find a hotel, you never book a hotel, that's a problem. Yeah, this is trying to help you figure out like find a hotel. Half of them are going to find train and then book a train and then forget about the hotel. They're not booking a hotel.

This is interesting. Okay, so this top row would be, it's 4 people trying to find a hotel.

Yeah. And don't get too caught up in this example. This is just the role play, so don't get too crazy about this data. There could be explanations for it. I'm just trying to like put you in the mindset of a data scientist. To say like, okay, look at the data. What do you see? Is there anything fishy? That's the main thing you have to keep in mind.

Yeah. So like find hotel, find train, book train. That kind of makes sense. I don't know why they didn't book the hotel as well, but yeah, no, that's the problem, right?

Like if they didn't book the hotel, then like, oh, okay. Why isn't that happening? We want to do error analysis on all the traces where there is a find hotel and there is no book hotel. And there could be other kinds of ideas that we have. We think about it. Maybe the restaurant situation is equally as interesting, potentially. If you refresh the page and then click on the matrix rows, I think it tries to— oh, that's cool. Yeah, that was a prompting the agent.

That's exactly why I like the multiplayer, because you could be like looking at stuff and then other people can be like enhancing it. Like, it's super fun. So I clicked on find restaurant.

So I don't know what's going on. I need to debug this data visualization a little bit. Looks like We go to find restaurant and we just book a train, but then that's it. So if we were working on this in real life, like I would tell the AI, let me click on one of the streams in the SandKey and then let me filter the trace viewer by that. And then let's start reviewing it. Like that would be cool. Right. And so it was very contextual. You might be thinking, yeah, let's create a SaaS. No, every dataset is so different. Like I don't do this exact thing for every dataset. It's like investigative work. We're just trying to explore and come up with hypotheses. So yeah, I mean, this one is like, okay, they're finding a restaurant, booking a train, and then that's it. If this is a product, maybe that's not good.

I'm guessing that for this product, anytime they book something is when you get a cut off the top, I guess. But I don't know exactly what the monetization is, or is it per use? I mean, obviously you want, probably want people to book, cuz that's some sign that it's useful to them. It'd be interesting to know, like, of all these people that do a whole bunch of finding of stuff but don't end up going to a booking, do they come back and book later? Because like a lot of times you plan a vacation or you plan something for weeks or months before you book anything, trying to see if you get time off. So it'd be interesting to see across those as well.

You could keep doing data analysis of all kinds. We could spend a whole day on data analysis of the dataset, just like within 15, 20 minutes. We can like already have some hypotheses here. And so you see like there's different kinds of ways of doing error analysis. One was like, okay, We just started looking at the data. We found some things by looking at the data we wouldn't have figured out just by doing data analysis. Even if we threw an LLM at it, we did fancy clustering, we wouldn't really have discovered that this is a bad user experience, I don't think. So there's different angles at which you should think about it. The most straightforward one is start looking at the data. Getting intuition of how to look at data, that takes a lot of practice. It's good to collaboratively sometimes think about data, think about what you're doing, especially when you're trying to figure out what a product does. Is it good? You might want to do that with your teammates, bounce off each other like Isaac and I just did. You might have different ways of looking at things. Isaac's building something that's like very built upon this collaborative, like you can have a meeting and do this collaboration thing with agents. I think it's using Claude code type SDK or something like that, potentially. So it's very familiar. You know, at some point I might do it. I might use a product like that in my consulting once it gets further along, but you just wanted to show it to you what it feels to like look at a dataset, work on it with someone, do some error analysis, just show you what it might look like. That's the point of this.

Yeah. And I think what's one thing that I really like is like, it's helpful. I think for like me and Hamel, we could both write code. We're both very comfortable with cloud code locally. We could do screen share, but I think where it gets more useful is if you've got like a product manager meeting with somebody else or a couple of people on the call, and a lot of people don't use a coding agent really at all, or don't really know how to use AI very effectively, or every time they do it, it doesn't feel good and they're not productive. And it kind of lets still collaborate and bring those people into the actual building of the annotation and the error analysis, which I really like.

Yeah, it's really cool. Cause if you don't like the annotation app you're doing, if someone doesn't like it, you can just immediately change it on on the fly like we did and just keep iterating. So it's pretty cool. As you can see, like, Isaac and I were working on it together, the application. So it could be pretty useful for you to think about depending on how you work and what kind of people you work with. Thank you, Isaac. This was really good. It was really interesting. Excited to see how this tool evolves.