LLM Eval For Text2SQL

I'm gonna do just like a few slides to sort of set the stage on, you know, some Braintrust stuff, and then just talk about how we think about like how to build really good evals. And then we'll jump into a demo where we just kind of walk through step by step of how to do a bunch of the kind of classic flows in building good evals. And then I'll share the notebook afterwards. So yeah, just a little bit about Braintrust. We're a startup company. We come from a variety of different backgrounds. Actually, prior to Braintrust, I led the AI team at Figma, and before that I started a company called Impura. And at both companies, we were shipping AI software. At Impura, we were, you know, in the Stone Ages, pre-ChatGPT, and, you know, training our own language models. And then Figma, we were building on top of you know, OpenAI and other models that were pre-trained. But the problem was, was the same. It was just really hard to make changes to our code or our prompts and know, you know, whether we had broken stuff and what we had broken. And so we ended up building internal tooling at both companies that basically helped with evals, and that is what turned into Braintrust. So You know, you'll see some of that today. The only other thing I'll mention, we're really fortunate to work with some really great folks as team members, investors, and customers. I think one of the really fortunate things for us is we've had the opportunity to build brain trust alongside the AI teams at some of the best product companies in the world that are building AI software. And so You know, something I take a lot of personal pride in is that a lot of the workflows and UI and just little details in the product are informed by, you know, some of the engineers at these companies who are giving us feedback all the time. Okay, enough propaganda about Braintrust. Let's talk about evals. So I, you know, I think everyone in the course probably already knows this. You've already learned about evals, but just like a very quick recap, why do evals in the first place? I think some of the key takeaways that you want to have when you're doing an eval and maybe the bar that you should hold yourself or the tools or whatever to is, you know, can you actually solve these goals when you do an eval? Do you want, can you figure out very quickly, did the change that I make improve or regress things, and if so, what? Let me like quickly and clearly look at good and bad examples and just, you know, use my human intuition and stare at them so I can learn stuff. Let me understand what the differences are between what I just generated and what I generated before so I can, you know, build some intuition about maybe what changed to the prompt, uh, led to that. Um, and then, you know, very importantly, we don't want to play whack-a-mole. Like, we don't want to improve some things and break other things and then, you know, not really know what we broke or why we broke it. And so it's important to actually systematically improve the quality of a system. Um, so, uh, uh, you know, the way that we suggest thinking about evals, whether you use Braintrust or, or not, is to think about it as basically, you know, there's 3 components that go into an eval. The data, which honestly, when you're starting out, and it's what we're gonna do in a second in the notebook, is you should just like hardcode it. There's all this fancy stuff you can do. You know, you can generate data, you can source it from your logs, you know, all this other stuff. But when you start, you might as well just hardcode it and sort of get going. A task function, which is, you know, the thing that takes some input and generates some output. You know, this is obviously a silly example, but it could be a single call to an LLM. It could be a RAG pipeline. It can be, you know, an army of agents that are talking to each other for a while and then coming up with an answer. But at the end of the day, it's, you know, taking some input, returning some output, and then one or more scoring functions. And there's so many different ways you can score things. Actually, in this example that we walked through, we're going to like handwrite some scoring functions just to really understand how they work. But there's a whole world of using, you know, LLM-based scoring functions, fancy heuristics, and so on. And so again, like, honestly, if there's one thing I could leave you with, it's, you know, evals are, you know, equal, equal, equal to just these 3 things. And it's important because sometimes I talk to people who are sort of lost in analysis paralysis about, you know, where to start. How do I, you know, I ran an eval, doesn't look great, what do I do? You know, it's important to just remember it's just these 3 things. So if you run an eval and it doesn't look good, yes, it can be challenging to sort of figure out what to do next. But it is literally just one of these three things. Either you improve the data, you change the task function, or you change the scoring functions. And there's actually a few different things you can do in each of these. I'm happy to send these slides out as well afterwards, but we're gonna, we're gonna actually go through a good chunk of these things in the demo today. But, you know, on the data side, there's, there's You can handwrite cases, you can generate cases, and you can get them from your users. That's pretty much it. There's, you know, maybe some other things you can do, but generally those are the 3 methods to, to follow. We're actually not going to spend too much time on the task function part of it. I think there's probably other material in the course that covers how to do various fancy things with prompts. We're going to do some basic things in today, and then on the scoring side, Again, there's really only 3 things you can do. You can write heuristic functions, which we'll do. You can use an LLM to sort of compare and do some reasoning about an output or an output versus an expected value. And then you can use your human eyeballs to sort of look at something and get an understanding of whether it's good or bad and then take some action on it. So with that, I'm going to switch over to walk through a notebook. Let me just see if there's any questions so far. There's no sound. Yes, there is. Okay, I hope the AV is okay.

Sorry, AV is okay. So you've got the 3 panelists, we can talk, and then we've got 230 people who can't talk.

Okay.

At some point you'll see there's a bunch, they're dropping questions in a Q&A thing, so we'll come to those questions in a bit.

Okay, perfect. Let's run through a real use case of doing some text-to-SQL stuff. I don't know if other people are watching the NBA, playoffs, but I'm a big fan, so I found some, you know, basic NBA data on Hugging Face, and we're going to use that to play with some text-to-SQL stuff today. So nowadays it's actually really easy to just grab some data. I think I actually saw a blog post this morning, and now it's even easier to load Hugging Face data into DuckDB. But I think it's a great way to play with these kinds of use cases. DuckDB is very expressive SQL tool, and it's easy to run inside of a notebook. And then obviously Hugging Face has so many good datasets that are available to play with for free. So, you know, this is the flavor of the data. It looks like it's probably one row per game, and I looked at this before and I think it's 2014 to 2018. So, you know, here's, here's the data. We're going to do something super simple to to actually do the text-to-SQL stuff. So just a little bit about what's going on here. This is the standard OpenAI client. I guess Hamel Husain sort of alluded to this earlier, but you know, in the spirit of staying simple, we, we really believe in sort of writing framework-free, simple code that just uses the OpenAI client directly. You can use all kinds of amazing tools as you get more and more into a use case, but almost always we see people start simple. Um, and so, uh, this is just the standard OpenAI client. Um, we did a little fun stuff here. Um, this base URL thing, uh, it's totally optional, but we're, we're actually going to use, um, this proxy, uh, that's kind of like a free thing that, uh, Braintrust has. It lets you cache LLM calls, uh, so when you're doing repetitive stuff like this, it's very helpful. Then this thing, we'll see it in a second, but basically it adds a little bit of special fairy dust into the client so that it traces the LLM call in a way that makes it easy to debug. But it's just the standard OpenAI client. We're going to grab the schema from the table. And then we're going to use GPT-4.0 for at least most of today. And that's it. You know, here's the simple prompt, nothing too fancy, asking it to write a SQL query. And let's give it a go. Cool. So here's the query that it generated. I don't know, Dan, does that look right? Well, we'll find out in, I guess, in a few minutes. But, you know, maybe. Um, let's, uh, you know, let's run it. Okay, uh, well, as a big Warriors fan, um, I definitely believe that that's right. So, uh, it looks like the Golden State Warriors won the most games in this, uh, period of time. Um, and again, you know, that's probably right. Um, so, uh, as we talked about, an eval is— it's literally just 3 things. Data, task function, and scores. We've pretty much implemented our task function. You know, this thing can take an input and then generate a query and a result. So we're pretty much there on the task function. We just need to figure out the data and the scores. Okay, so to create the initial dataset, I just wrote 5 questions. Honestly, I'm too lazy to write the SQL queries and the answers, and that's okay. I would encourage you to be lazy as well in certain cases. And so let's just use these questions. And what we're gonna do is actually to start, we're not going to evaluate whether the model generates the correct answer or not. We're just going to evaluate whether it generates an, a valid SQL query. And then we're gonna do some human review to actually bootstrap a dataset from there. So we'll just use these questions. As I mentioned, we basically already wrote the task function. So this thing just, you know, takes those 2 calls and puts them together into one function with a little bit of error handling so that we can, you know, distinguish between a valid and invalid query. Then the scoring function, let's just keep it very simple. We don't really know what the correct answer is to these SQL queries yet, so let's just see. Let's say it's a good output if there's no error and it's a bad output if there is an error. Again, to Hamel Husain's point earlier, very little framework stuff in here. Just, this is literally just plain Python code. Let's keep it very simple. And now we're just going to glue this stuff together into an eval call. And, you know, we give it a project name. The experiment name is actually optional, but it helps, you know, keep— it helps us keep track of some stuff easily. And then we'll give it the questions the task function, and the score. And we can run that. There we go. Okay, so let's take a look at this. Okay, so welcome to Braintrust. Let me go into— let me actually just make my thing light mode. There we go. Okay, welcome to Braintrust. So you can see, you know, really quickly, it looks like 3 out of the 5 passed this no error thing. We can just quickly look at these to get a sense. Looks like there was a binder error and looks like a binder error over here. If you recall, I mentioned that OpenAI wrapping stuff It sort of nicely captures the exact LLM call that ran. So you can see, you know, this is how we formatted the columns. That looks about right to me. If you wanna mess around with it, you can actually, you know, do that. You can rerun the query, you can try out different models and stuff. We're not gonna do that right now, but just to give you a sense of some of the exploration you can do. And then let's actually look at these ones that were correct. So Let's see who led the league in 3-point shots. Pretty sure Houston is correct. That was James Harden's sort of golden era. Okay, great. So what we're gonna do is actually, again, we were lazy earlier, but the model has very kindly for us generated a good piece of data. And so we can actually save this as something that we use as a reference example. And so I'm just gonna add it to a dataset. Let me make sure I cleared this out properly from earlier. Yeah, perfect. So I'm gonna add it to that dataset called golden_data, and we'll use that back in our code in a moment, but let's just look at these other ones. So which team won the most games in 2015? That's an empty result, so that, that's clearly not the right answer. And, you know, my spidey sense tells me that this is probably not the right format for the date, and it's filtering out rows that it shouldn't. So, you know, I don't know, Dan, probably we should give the model a preview of what some of the data looks like so that it knows what the right date format is, for example. So let's just, let's just keep that in our heads for a second. And then this one, again, big Warriors fan. We know that this is correct, and so we're gonna add this to the dataset as well. Awesome. Let's just quickly look at the dataset. So it's kind of the same thing as an experiment. You know, it has some columns. We can play with the data. But now what we've done is we've actually bootstrapped some golden data, and this is, data that not only do we have a question, but we have a reference answer, both the SQL query and the output that we believe to be correct. And because we, we have that, we can actually make some stronger assertions when we're running an eval to compare the query to the correct query and the generated answer to the correct answer. So let's go back and what we're gonna do is rework a few things to take advantage of the golden data that we have and also try to fix some of the issues that we saw when we were poking around. So I'm gonna change how the data works a little bit. Now I'm going to read the dataset from Braintrust, and I'm going to return those golden questions. And then the remaining questions that are not in the golden dataset, I'm just gonna have the question itself. So now you can see for the, for the things that are in the dataset, we have this kind of fancier data structure which has the question, It has the expected answer. It has some other stuff that is— Braintrust will take advantage of. We don't need to bother with it right now, but it's really those, those things that matter. Okay, cool. And now let's change the prompt a little bit. So instead of just looking at the columns, we're going to get a row of data and then sort of inject it into this table. I really hope I formatted this correctly. We can double-check. That I did in, in a second. But, you know, it doesn't matter too much. It's okay to be wrong every once in a while. And so let's do that. Looks like— okay, that looks like a much better thing. You know, it's probably taking advantage of the date format that it knows about now. And then let's improve the scoring stuff a little bit. So we're gonna actually pull a few fancy scoring functions from AutoEvals, which is an open-source library that we and our sort of community of customers and users maintain. And we're gonna add 2 more scoring functions. So this correct_result function is going to get all the values from the query. We don't care too much about the column names, so we're just gonna look at the values and then it's gonna use this JSON diff method to just sort of recursively compare them. There's probably some better stuff we can do, but let's just start here. And then we're also going to use a, an LLM-based scorer called SQL. And this actually asks a model to look at the reference query and the generated query and try to determine if they're semantically answering the same question or not. So we'll just save these.

Is that score— is that score binary, or how— what's the score mean, or what does the score look like?

Let's, let's look at it in a second. It's a good, great question. Um, so, uh, let's just, you know, plug it together. So again, we have this eval thingy. We haven't— we redefined this function. Now we're using this load_data thing to get the data, and now we have these 3 scoring functions. So we'll just run this. Awesome. Okay. So let's start by answering your question. So anytime you run a scoring function in Braintrust, we actually capture a bunch of metadata about it. We really, really believe that debugging a scoring function is as important as debugging the task function itself. So let's, let's, let's look at like exactly what the model sees. So this is the prompt that the model saw. It said you're comparing this. Here's the question. Here's the expert SQL. Here's the submitted SQL. This is the criteria. And then it's actually doing a function call under the hood. And the function call is just picking one of these two values, correct or incorrect. And so this is a binary score, basically, that's asking it to do that and sort of explain its reasoning, which you can also debug right here. Great. So let's sort of analyze this as a whole. The first thing you'll see is that Luckily, we did not break those two that we know to be correct. Like, we didn't break the SQL queries or the results. It looks like we improved on the no error front. So there's one example that did not return an error that did return an error before. We can actually, if we wanna zoom in on that, we can just look at that example. It looks like, you know, before it returned an error, Now it didn't. If we turn off diff mode, we can look at this, determine whether this is the correct answer or not. I actually don't think it's the correct answer because it's looking at which team had the biggest difference in consecutive years and it's hardcoded 20 and 19. I don't think this is the right thing. That's one for us to improve. One thing you could do actually at this point is you could add it to the dataset, and if you want, you could try to handwrite the correct SQL query and the answer. I'm not gonna do that, but you could. And let's just poke around a little bit more. So that's still an error. Which team won the most games? This one was still correct. Again, I, I like really, really like to manually look at data because it allows me to double-check all of my assumptions. I don't trust anything, and so I spend a lot of time actually doing this when I'm evaling stuff. Which team won the most games in 2015? Okay, this is the one that had an incorrect answer before, and now it looks like it's actually correct. You can turn on diff mode and you can call that it returned an empty results before, and now it returns something. So again, you know, as a Warriors fan, I know 100% in my heart that this answer is correct. And so I'm going to add this to the golden dataset. Awesome. So now we're, you know, we're building up a dataset. I think we're like 15 minutes in. In my experience, honestly, 50 good golden examples that have been lovingly curated by a passionate small team of people is really all you need to have a really good AI application. We're already like a good chunk of the way there, just sort of playing around with this demo. But, you know, hopefully it's, it's, it's kind of clear. It's not, it's not crazy rocket science to do this stuff. I think it just requires some care and like, you know, looking at the data closely. Um, okay, cool. So, um, you know, there's a bunch of stuff we could do from here. We could try to, um, play around with the prompt a little bit more. Uh, while I was building this, I was thinking about maybe like feeding the errors back to the prompt, uh, in a new message and asking it to generate another one. Um, but just, uh, to take it in a different direction, like I said, there's like 3 things you could do. You can change the data, You can change the task function, you could change the scoring function. Let's keep playing with the data. And what we're gonna do is actually try to generate more data than we currently have, and we're gonna use a slightly different method. So the first method, we kind of handwrote some really good questions that we thought pinpointed at the, at the task that we're working on. What we're gonna do now is more of like a coverage style thing where we actually use a model to generate questions for us. Um, and, uh, this code, again, it's not that much code. Um, so I'll just walk through it really quickly. We're going to use function calling to sort of force the model to generate, um, stuff that we can easily parse. Uh, and so in this case, we're going to have it generate a list of questions. Um, and each question is going to contain SQL and a question. Um, I'm specifically asking it to generate the SQL because my intuition is that if I have the model generate SQL and then generate an explanation about the SQL, it's more likely that the explanation is correct than if I have a model generate a question and then a SQL query that answers the question. It feels less likely that it would generate the correct SQL query. And so I'm kind of asking it to, to do both at once and really focus on the SQL. Then I'm doing my, you know, same sort of prompt as up above. Um, you know, honestly, a fun thing you could do— we're not going to do this right now— but you could, um, try using a non-OpenAI model here just to sort of, um, avoid some, uh, built-in biases that one vendor's, um, models may have. I'm, I'm not going to do that here, but You, you could, um, or you could do both. Um, and yeah, I'm just going to ask it to generate some queries, uh, give it the schema, and that's it. So we'll run this. Um, here's an example of a query and the question. What do you think, Dan? This looks about right to me.

Having trouble. Uh, are they asking for 82? Are they asking for the 82 times the number of teams there are? Oh, they're doing a group by team. Okay.

Yeah, yeah, yeah, yeah. Cool. Cool. Okay. Um, and the last thing we're going to do is my guess is that some of the queries that it generated are bogus. Uh, like they're just not, not valid SQL queries. So we're just going to try them, and then the ones that succeed we're gonna add them to a list. So let's do that. Okay, cool. Looks like a few of them didn't work. And then here's an example of one that did. And because we generated it, we have this really rich data structure that, that we can use. So we, we have the, the query or the question. We have the, the expected results because we run the query. And we have the SQL. So that, that's awesome. Now we're gonna add this to our load data thing as well. We're gonna kinda do the same thing where we load the golden dataset and then we get all of the questions and generated data that are not in the golden dataset. The, the reason I wrote the code this way is that now as we keep iterating on this, each time we add something to the golden dataset, if it overlaps with the generated data or the original questions, we don't need to sort of worry about that. We don't really need to change our scoring functions 'cause we already have scoring functions that test the correctness of the SQL, whether it's a valid query or not, and whether the results are correct. So let's just reuse the scoring functions that we had before and run another experiment. Awesome. Great. So you can see this experiment is automatically compared with the sample one that we did. We could compare it to the initial one if we want. It's very easy to play around with this. Looks like we didn't regress anything. There's not a lot of overlap between the two, so this makes sense. We didn't change the task function at all, and we just added data, and so makes sense that we didn't improve or regress anything compared to the previous experiment. And it, it looks like we have a fairly rich set of, values here. So, you know, there's some examples like this one looks like it was one of the generated queries and looks like although we have an expected answer here, maybe the generated SQL query is a little bit different. And so, you know, we should dig into this and understand which one is correct. Maybe the generated answer is incorrect. Maybe the original data generation process yielded something incorrect, but at least we have something to diff and sort of understand that. You can also sort of break this down here. So if you want to, if you want to look for a category and try to understand what the differences in scores are, that's really easy to do. Or if you want to break it down this way and actually see, like, of the 3 categories, how did we do? It's easy to sort of explore the data and just look at the subcomponents that you want. And so, you know, from here, again, we've, we've just made our eval process richer. Now we have more data. We could go and generate even more data, just literally run that code again and generate 10 more examples. Try a different model. We could explore some of these cases and try to understand why is essentially the model disagreeing with itself here, because we use GPT-4 in both cases. So this is probably a really good example to dig into. Or we could just try changing the task function, maybe provide it 2 rows of data, or maybe provide it some summary statistics about the minimum and maximum year or something to help with some of the questions. Um, there's so many different directions that we could take it. Um, however, just for fun, we're going to take it in kind of a simple direction, which is let's just try GPT-4 and see how that does, uh, compared to GPT-4.0. Um, so I'm just going to change this global variable which I referenced above and give it a go. Interesting. Okay. So it doesn't look like it was a slam dunk. You know, we actually— it looks like regressed across the board with GPT-4 versus GPT-4o. I would personally be curious I guess we can sort of look at this quickly. Like, does this— how does it vary between— yeah, it looks like we even regressed on the golden data. So the golden data we know is correct. The generated data we're a little bit less certain on, but it looks like it even regressed on this. I mean, I would actually be— I haven't looked at this myself yet, but it'd be kind of interesting to understand. Yeah, it looks like, you know, for example, GPT-4 messed up the date syntax once more. We should make sure that we gave it the right prompt. Yeah, it looks like we gave it the right sample. I mean, this formatting could definitely be improved. It doesn't look like my newlines maybe made it through correctly. But yeah, this just kind of gives you an idea. There's so much stuff to do. In digging through data and understanding it. And, you know, literally starting from nothing, just a dataset. I think we've, we've kind of iterated our way into— and sorry, no dataset, no eval dataset. We just had like an NBA dataset. We've iterated our way into a pretty rich application here where we have, you know, a non-trivial prompt that's generating queries. We have a pretty rich dataset that ATT&CK different parts of the problem. And then we have 3 scoring functions that help us diagnose systematic things like, is the query even succeeding in the first place? From some more nuanced things like, does it return the correct result? Or is it semantically the right SQL? So there's a lot of different places to take it from here. And that's why—

I just want to just point out that for students watching the class, This is actually very similar to the Honeycomb example that we've been talking about in many ways. Like, you know, my workflow was actually very similar to this. We didn't have any— we barely had any data to start with. We had to synthetically generate lots of data. You've seen that. And so I think it's really interesting here. Like, you can see, like, what a workflow might look like if you use a tool and how, you know, it might automate some things and help you organize all of this information. So it's actually like, this is not, this is not necessarily like a toy example. Like, this is actually like very closely mirrors like something that I've done in real life. And I think it's great that Ankur is like, you know, kind of doing the SQL use case because it maps really nicely to the one that you may have been practicing with already.

Awesome. Yeah. And I'm very happy to share. We'll publish the notebook. right after this so that folks can play around with it and, you know, tweak it and, use it in their own environment.

We're going to have a bunch of questions, but I have one right now. Um, so there is a bunch of functionality built in if you're generating SQL that is really, like, cool and is sort of taking advantage of the specifics of knowing that you're using SQL. What are the other type— I think we saw something for generating JSON. What are the other types of use cases or generated text where you've got some magic built in?

Yeah, I mean, um, the, uh, you know, the only thing that we use that was built in here that's SQL related is that one scoring function that, um, compares two SQL queries. Uh, we have, uh, in auto evals, um, we, we sort of optimize for quality over quantity. So I think there's about 20 different scoring functions that help you out with things like RAG. And, you know, within RAG, assessing the answer, the generated answer that, you know, retrieves context relative to the answer, relative to the query, and so on. There's a really popular framework called Ragas. We have actually an implementation of the Ragas metrics built into AutoEvals with a few bug fixes and, you know, uses function calling to improve the accuracy and so on. We also have a bunch of tools for measuring the output of tool calls, and we see people do increasingly like complex agentic tool calling workflows where you're measuring both the output of individual tool calls as well as kind of like the end-to-end flow. Did you accomplish the task that you initially wanted to set out to do? And then honestly, we have a bunch of just building blocks that are— they sound really simple, but having written a good chunk of the code myself, they're like, they're just a pain in the butt. Like, comparing two lists of strings in today's world is really hard. Um, uh, first of all, like, normalizing that comparison to a number between 0 and 1 is not trivial, even if you're just comparing the strings. But I doing it in a way that is semantic is even harder. And so we have one of the most popular scoring functions is a list of strings comparator, and you can plug in the comparison function. So you can do like GPT pairwise comparison, you can do embedding comparison, you could do, you know, Levenshtein and so on. So just some building blocks like that that I think are very painful to handwrite, but generally quite applicable. Cool. Awesome. Yes, I see some questions in here. Do you guys want to MC through the questions, or should I sort of read?

I'm happy either way. If you're going to do it yourself, you should sort by most upvotes.

Okay, why don't you MC them just because make it more interactive.

Great. Uh, we got two from Wade Gilliam. Uh, Braintrust looks a lot like Langsmith. Wondering what the feature differences are or pros and cons from experienced people. If you were saying like, when should someone prefer one or the other? What are the—

no, feel pressure to answer this question. I know certain— I know that there's some culture around, hey, let's not, you know, try to attack or try to compare ourselves to other vendors. So feel free I mean, you don't have to answer it directly.

Yeah, honestly, I think, um, uh, I will, uh, hold off. I, like, I have a lot of respect for the team at LangChain and, and all the amazing stuff that they've built. So I, you know, I'm not going to say anything bad about, about their product. Both products are pretty easy to try. Like, you can just sign up on the websites and try them. There are a number of differences in the workflow and the UI and a bunch of other things. So I would just, if I were you, I'd just try them and sort of get a sense of what feels right for what you're trying to do. I know Hamel Husain, you tried like 25 different tools or 30 different tools. If he can try 30, you know, you can try 2. And so you might as well.

More like a dozen. I'm not that crazy.

Okay.

Okay.

Well, sure.

It felt like 30, but yeah.

Cool. Next question. How would you run open source or OpenWeights models in Braintrust?

Yeah, great question. Let's actually answer this sort of specifically. So again, I know I sound like a, you know, like a dead horse or whatever, a broken record, but evals are— they're just 3 things: data, task function, and scoring function. So, um, let's talk about each of them. So data, you know, we just handwrote these questions. There's no— doesn't matter what— it's no OpenWeights or closed weight model, you know, it's just these. You can do the data generation however you want, it's just a script, um, later here as well. Uh, the task function, um, so we have our generate query thing, um, you know, none of this is OpenAI-specific. This is— and, and all Braintrust cares about is that you write a function that takes some input and returns some output. What happens inside of this function, it doesn't matter. Um, if you use the OpenAI client library, then you get some nice tracing features. Like, you get inside of, uh, Braintrust, you get, um, you know, this kind of like really nice formatted thing But it doesn't— you don't have to do that. And most of the OpenWeight models hosted on platforms like Together, for example, they actually have OpenAI-compatible APIs. And so you could do that and get sort of like all the tracing goodies and stuff too. I know someone asked about caching. The proxy is actually an open-source proxy. Project that we maintain that's deployed on Cloudflare, and you can deploy it in AWS or, you know, your own Cloudflare, wherever you want as well. It, uh, in each place that you can deploy it, it just uses like, you know, whatever the easy durable key-value store is. So Cloudflare has its own key-value store. Um, it end-to-end encrypts the payload and, um, response in terms of the API key or rather a hash of the API key. And so, um, it's, you know, very safe from that standpoint. And, um, the proxy is also pluggable. And so you can point it at self-hosted models, or you could even point it to a model that's running on your laptop. I, when I'm on long flights and I'm working on this stuff, I run Ollama locally and then point the proxy at it. Um, and so, uh, you can also use kind of that abstraction with an open weight model, and many of our customers do.

We've got a couple questions about sharing the link to the notebook. I think that may have already been shared in the Discord, but at some point, uh, I assume that you'll— are you comfortable sharing a link?

Of course. Yeah, yeah, yeah. I'll just— uh, we actually have this section on our website, um, called the Cookbook. And there's a bunch of use cases here. I can share this in the Discord channel. I haven't published this new one yet, but I'll publish it in like 2 minutes after we, after we wrap up this call, and then you'll be able to sort of scroll through it like this and then access the notebook here.

Well, um, how about this one? So we've got from someone, an anonymous attendee wrote, I love this workflow. Are there examples or tasks that you'd say do not lend themselves to data generation of the kind that you showcased today? Or maybe are there tasks that you sometimes see people ask you about and you're like, ah, we're not a great fit for that yet?

That's a good question. Let me try to think of some recent examples. You know, I, I think one thing that comes to mind is if you're doing classical ML, I think the shape of your data is quite a bit different. So let's say you're doing like a fraud classification problem and you're training like a boosted decision tree and you are, you know, you have like 1 million examples that you want to test, which is totally reasonable because it takes less than 5 milliseconds to run on one example. I think that the workflow in Braintrust theoretically will work. However, as you probably saw from me clicking around, we, we really believe that it's important to stare at individual examples and build a lot of intuition around them. And so a lot of Braintrust is looking— it's helping you wayfind to individual examples that you can look at in more detail. Someone asked about the difference between us and other products, and, you know, spiritually, I would actually say that's something that is very informed by my personal experience working on evals for like 8 years. It's just— that's just the workflow that I've really done and refined. And I think that's actually pretty unique to Braintrust. And so, you know, in LLM land, I think wayfinding is really important. In classical ML, I think looking at aggregate statistics with multidimensional visualizations and stuff is actually often a more useful way to analyze the data. And so that's probably a use case where I would recommend using a different tool.

And, um, I assume— two questions. Um, I assume that you're looking for entirely text data and that you don't— you are not used to someone submitting images or getting images back.

Is that right? No, we actually support images, uh, natively. Um, we also support them in our Prompt Playground, uh, and, you know, we visualize images. LLM calls show images and everything.

Cool. And then do people, um, when they're using Braintrust, is that purely during model development, or do they have some Braintrust call to collect data on a deployed model?

Yeah, um, so we have— there's two major use cases for Braintrust. One is what we would call offline evals, which is what we talked about today, and the other, um, It's sometimes called online evals or observability or logging, um, and pretty much all of our customers do both, uh, and they integrate really nicely together in Braintrust. So, um, I can actually show you, uh, just really quickly, um, maybe we can kill a few birds with one stone here. Um, but here's an example where it's actually not just generating an image, it's generating HTML, and you can render, you know, the HTML that's interactive, uh, right here in Braintrust. There's a sandbox built into the product because a lot of our customers are generating UI, uh, using, uh, AI. Um, and this is the logs view, so it's actually not different, not very different from the eval view. Um, it's another thing that, uh, makes Braintrust, I think, quite special and different is kind of the very tight integration and identical data structure between logs and evals, and you can capture user feedback right in the UI. This, you can capture it through the API from your users. Um, if your users leave comments, you can actually capture those too through the API, or you can just save stuff in the UI, and then you can also do the same thing where you add stuff to a dataset from here. Cool.

I got a couple more. Top one right now is, often when we develop text-to-SQL applications for SQL database, there are a bunch of challenges due to lack of familiarity with the database. In our case, there's no documentation available and the database is large and complex. This complexity makes it difficult to determine the right question to ask because you don't have a clear understanding of the database's structure. Do you have any suggestions for operating when you've got a complex database and you don't actually understand the structure of it?

Yeah, I mean, I think that is a great question because first, in practice, things are much hairier than what we can talk about in like a 20-minute demo session. But honestly, I wouldn't overcomplicate it. Like in those scenarios, I think the most useful thing you can do is, you know, let's say that you're doing that in the context of like an internal tool that your teams can use to do, to ask questions on some data. I would instrument that tool from day one to generate logs like this and capture user feedback and then try to categorize the questions either using a model or asking your users to categorize them or categorizing them manually. You know, it's not the end of the world to do that every once in a while. You can tag them or, you know, do whatever is easy. And then what, what I would personally do, if you remember over here, we sort of built these different subcategories and then we looked at the scores within the subcategories. I think you want to sort of get to a point where you can successfully classify the different types of questions, whether it has to do with like the nature of the question or the tables or subset of the schema it's targeting. But try to build an understanding of, you know, what kinds of questions you can answer successfully and which kinds you can't. And that's not going to happen overnight. You need to sort of iterate towards it. But getting to a good taxonomy is very, very valuable. It's a little bit of an art form. So that's where a lot of your creativity can, can come in. And then, and then you, then you know exactly what to do. Like either you say, hey, there's a set of questions, like I have my super complex data and none of the financial questions or auditing questions work. Maybe we focus on improving those for a while, or we accept that the dataset that we are running the Text to SQL app on is inherently too messy to do, um, auditing questions. So let's do some schema work, like let— maybe let's create some views or something that make the life of the model a lot easier and see if we can improve that category.

Yep. It's also nice, uh, it seems like a place where you see the connection between what you were showing for observability on a deployed model, and maybe you even have like thumbs up or thumbs down Coming back and looping back to the experimental, uh, yeah, exactly, exactly.

Yeah, there's some stuff too, um, uh, you know, for example, I'll show you, um, really quickly, like, you— we also make it really easy for you if you hit the R key from anywhere in the product, you can actually enter this sort of human review mode where you can just really quickly use keyboard shortcuts and actually like rate things. And this is really good, especially in a use case like that. If you have maybe an analyst audience who could look through a bunch of questions and answers and just rapidly rate them, I think it's, it's really powerful to do that. And you don't have to like pre-populate a queue or do anything like that. You literally just hit the R key from anywhere in the product, log view, experiment, etc., and you can sort of enter this workflow. Cool.

All right, thanks so much.

Thanks for having me.

All right, see ya.