Welcome to our live show!

To have you here. So today we have Jim from OpenAI who works on, among many things, he also works on evals at OpenAI, specifically applied evals, and helps build those products that help customers with their applied evals. And so I'm really excited. We're gonna hear some case studies and some best practices. I'll go ahead and hand it over to you, Jim, to kind of intro yourself and tell us a little bit more about what you do.

Thanks. Yeah, absolutely. And let me get the screen going here. Yes. Thanks for inviting me. I'm really excited to talk about, about evals. It's an important part of the customer process. Hopefully, you'll come away with some action items here. Thanks for the intro. I manage the model customization team in the API group. We help customers get the best out of the models that they have. That means everything from evals, as you pointed out, to determine what best means, to following up with things like fine-tuning and prompt iteration and anything else you need to get the best out of the models.

Um, and how does someone avail themselves? Like, how do you— is there like some requirements that you need to be able to collaborate with you on model customization?

Yeah, great question. So the evals product we have and the fine-tuning product we have are generally available, meaning anyone can come and, and fine-tune and, uh, and have those fine-tuned models available in the same way that you use, uh, OpenAI models in production. For collaboration, we really look for folks, uh, and actually we'll touch on this, we really look for folks who have a really clear idea of what the model gap is, uh, and the most effective way of communicating that, unsurprisingly, is evals. So typically you'll see some case studies, um, but a customer will come to us who has a good amount of usage, and so we can see that there's a lot of value we could provide to the company and to their end users. And they'll say, "Hey, we have this gap." And if we can't advise through our forward-deployed engineers or customer success, we'll often engage directly and start developing new methods, like our recently launched preference optimization, to be able to, like, serve the needs of those customers. And so we do that in a pretty organic way through an, alpha and beta process, but it usually starts with evals.

That's really interesting. And I mean, do people know how to do evals? Like, do you find that people come to you with pretty good evals, or do you have to help them with the evals?

Great question. The short answer is no, people don't know how to do evals generally, and it's something that our engineering and, and forward-deployed engineering team helps a lot with. Everything from starting from, you know, a very basic what are evals and why are they valuable to often helping build out those first cases for a company. So many of the companies that I'll go into, several of them started with very little eval infrastructure, including the first case we'll talk about, Morgan Stanley. There are some companies that come with a deep understanding, often companies that have a longer history and background in ML and pre-AI functionality. But, but the larger kind of functional organizations that are looking to improve existing workflows and that sort of thing often need a crash course. That's why I'm so excited to talk with you, honestly, is I think you are doing, uh, kind of God's work here in spreading the message on why these things are important and, uh, improving the, the overall industry's understanding of, um, of these sorts things.

Okay, great. Yeah, let's, let's get into it.

Um, sweet. Yeah, I'm gonna be talking about 3 key takeaways I hope everyone can, can walk away with understanding and, and some examples of, of how these customers exactly made these, these ideas true. Um, so just jumping into the first case, I mentioned Morgan Stanley already. Um, they are one of our, uh, first partnerships. They are still one of our, our strong partners in, in adopting some of our cutting-edge stuff and One of their early cases was an internal chat application that could search over their internal documents and return and summarize relevant information to financial analysts and advisors and, and provide their end customers with a lot of information. So one of the things that we first had to work with them on was an eval framework. The important part, I think, for this lesson is that you can start and should start very simply with evals and that these small handful of eval examples can already start driving big improvements. Along the way, you could build out that fuller, richer eval set. It does not have to start with a 12-dimensional spreadsheet that covers every corner case that you can think of. And so specifically, like, walking through those different steps, starting with the prototype and figuring out what you want to nail in your prototype example can be really telling in terms of the first several eval examples. These are the things that you want to always make sure to put your best foot forward.

And these are specific sort of, uh, stories that you want to nail, like specific user interactions. Is there any example of, like, kind of example of a specific interaction in this Morgan Stanley case that you can think of?

Yeah, I, I do. I remember one of the early, uh, examples that they gave is they wanted to be able to have a financial advisor compare the outlook of two different companies, you know, say Disney and Comcast, um, to figure out, um, what were the different aspects of those two companies with the up-to-the-minute data that Morgan Stanley had in order to advise in different ways about how, uh, they might invest, um, some of the money into the, those two different cases. And so that specific example, and I'm forgetting if it was with those two companies, specific two companies or not, but that specific example is one of the foundational places where, hey, we know roughly what type of documents we want to be pulled in. We know the type of key insights that financial analysts are looking for from these documents. And we know roughly like, the amount of text, whether it's a couple paragraphs in the common case versus a sentence or, you know, a full report is going to be useful to start. And so those sorts of specific examples were really important in the early stages, again, versus a more general— like you mentioned user stories. And so the key difference here from a user story and a specific example is that these had, you know, the companies that, uh, the, the product owner really wanted to nail in their demo versus a user story that might say compare companies A and B and like, and then, and the result being like share useful insights. The— I say prototype here intentionally because yeah, uh, we want to get into specifics when we want the demo, um, to be able to, to behave in the exact way that we're, we're hoping.

How many examples, just an order of magnitude, like how many we're talking about here?

Yeah, honestly, this is like a handful. It's like 5.

Okay.

It's really, it really is aligned with this idea of a prototype. You're sitting down and showing someone this for the first time. You're not going through every case, and hopefully your, your product idea you have, your model interaction, is giving a wow factor to the people who are sitting and watching the demo. And specifically, I wanted to talk about this as a simple short start because as an alternative story that I wanted to tell, there was a company who tried to go out and source 100 different examples of the things that they wanted to do. And honestly, you know, the first 5 were pretty good at like what the customer wanted to do. The next 10 were okay, like kind of in the same neighborhood, kind of made sense. And then honestly, the, like, the last 80% were just not very compelling, just iterations of the same thing, and, and frankly not very insightful. They didn't add very much to this, um, you know, first pass at evals. And so actually spending the time on getting the first 5 eval examples right was much more valuable than trying to go out and source 100.

So what I'm hearing is maybe because like it's a spectrum when you're first starting out, when you have your prototype, you know, you don't want to go directly into evals. You want to do— you want to go on vibes, honestly, in the beginning. It's very valuable to take some examples. Is this kind of what you're talking about? Like, hey, do the— make sure you get to the minimal place first before you put in a bunch of measurements.

Yeah, that, that's right. And I think I think going on vibes is a very common starting point for our customers as well. The key difference is write down the vibes. It's like, you know, have some mechanism of tracking them because I think what happens very easily is you go past the vibes, you start getting these real user behaviors, you start doing a limited rollout, and then you've kind of forgotten and or not tracking those initial cases that you want to make sure are right. And so simply writing them down and we can get into like different ways of doing that. In the Morgan Stanley case, they started with an internal tool that they built in order to do this testing with, with a data store and some specific examples. But in the general case, you could use tools like OpenAI's eval product or tools like Braintrust or PromptFu in order to, to keep track of these examples. But that's the big difference is you know, write, write them down.

Okay, so you start out with the Morgan Stanley. Okay, start out with vibes on these, like, like, you know, handful, 5 examples. And then I see on the slide, like, limited rollout. What does that mean? Like, um, is that still kind of this, like, comparative analysis, or did the scope increase? And what was actually going on here?

Yeah, so what was going on here, instead of going, you know, directly from vibes, uh, this looks good, to making it available to the whole department or the whole company. They chose limited teams, limited pockets to start getting more and more feedback. In the middle here is real user behavior, which we, which we touched on barely, but like that's when you show it to someone else in the hallway of your idea, really take note of how they use it. And that is a very useful, you know, you started with your first personal 5 vibes, but the way in which people initially use it really can give you a lot more sense that you're witnessing directly and talking to them about what they expect out of the product, and which you can use to expand your set to that next 10 or 20 use cases. And then the limited rollout, the difference there between real user behavior and limited rollout Limited rollout, you're not really supervising, you're not showing someone the laptop and understanding what they're trying to do. You're trying to collect larger-scale feedback. And the limited rollout step in your product, it's really important to have a way for users who are not sitting next to you to be able to still give you feedback. And I put the thumbs up and thumbs down here specifically because you often don't need a very sophisticated tag hierarchy of, of, uh, of cases, often just a simple thumbs up and thumbs down lets you sort through the feedback well enough that you can yourself get a sense of, of where the problems are.

So like this, just to clarify, it sounds like it's not like you're using the thumbs up, thumbs down as the final, let's say, uh, eval. You're kind of— it sounds like you're using it as a way to sample efficiently. After people give you a thumbs up and thumbs down, then you go take a look and you do, and you do the error analysis and you say, okay, like, what actually went wrong here? If you have enough domain knowledge, I, you know, it sounds like you can infer what went wrong and you're just like reducing the friction of giving this feedback, getting the feedback.

100%. Yeah, you got it. And I know you're a big, uh, proponent of going and looking at the data, and I 100% agree. Um, and there's no substituting that because understanding what the issues are— no, in one— in actually a few customer cases, we've had cases where the users were giving thumbs down to certain examples, but the reason they were giving thumbs down is often because the user wasn't using the product as intended or they weren't getting the functionality that they ultimately wanted out of the full experience. Variance versus the model giving back a bad answer. And so being able to kind of separate those pieces of feedback in the early stage of the product is really important.

And do you happen to know, for the error analysis, um, you know, was it like pulling this data, putting into a spreadsheet, writing notes, and categorizing those errors? Do you have a sense of like how it was done? Um, you know, what kinds of errors came out? Like you know, any memorable ones, like categories?

Yeah, great question. So in the Morgan Stanley case, the thing that popped out immediately was recall of documents. Um, and so, yeah, to your point, uh, instead of spending a bunch of time trying to tweak the model itself in the early stages, we spent time working with them on their vector database, being able to pull and recall enough documents, um, that they— that the model could then summarize and answer questions about. So even from a handful of these, these problems— and, and often in my experience at this early stage, you know, you can look at 5 thumbs-down, um, examples and start getting a pattern of like, oh, I see, like, we're not getting enough of the documents that, that That makes sense.

So like, you, you sounds like, okay, you got this, uh, signal, like, you know, thumbs up, thumbs down. You quickly identified, okay, the biggest problem is retrieval. Then did you kind of switch focus into retrieval and start with a bunch like retrieval search-based metrics?

Yep.

And how did that, how did that go? Like, how did you assemble that, uh, yeah, exactly.

Um, and so that, that was a pretty That was a pretty different work stream where we knew that recall, uh, and we'll talk about, um, you know, sequencing metrics here, but like we knew that recall was one of the important factors of user satisfaction in a query, and user satisfaction ultimately led to like being able to serve the customer request. But because we knew there was a gap there, we can zero in on the recall aspect of that, and we worked with Morgan Stanley on being able to improve the recall of documents, also improve the model a little bit on the, the types of searches that it was doing and the number of search terms that it would use to, to, to look for. Or in our case, yeah, instead of just search terms in terms of words, being able to use different embeddings in order to find relevant documents.

What were the main things you did to improve recall? You know, you know, yeah. What are the levers that you pulled there?

Yeah. And when you say you, I should also preface a lot of this by, you know, I'm representing the work of an amazing team, an amazing partner in Morgan Stanley, and any, you know, misrepresentations or misrememberings is purely my fault. But yeah, but some of the work that we did there were pretty typical, I think, of the, the, the RAG optimization process, being able to try different level of embeddings of different size of text. So do we embed the full document? Do we just embed paragraphs within it? When we do recall, are we trying to recall the full document that the paragraph was in, or do we just recall the specific paragraph? Does it have enough context So it was kind of playing around with those different aspects.

Okay. And you were able to— like, what was the feedback, like the iteration cycle? Like how fast were you able to get these metrics when you made a change, roughly?

Yeah, in the ideal case, we— when we, when we got this full process spun up, if I recall correctly, we were able to do kind of nightly or, well, daily changes that we could test each night. Which I'll go into here a little bit with the CI/CD stuff. But being able to tweak some of the settings and processes each day was where we wanted to end up. I won't lie and say we were there from day 1, but because we were able to get some of these tests written down and be able to automatically run them, we could start making tweaks each day. I think there are some small, smaller startups that we work with that make tweaks and changes multiple times a day. I think it ends up being a very similar process to more traditional software engineering iteration in which you always want to increase that, that cycle time, but it can be limited by things like re-indexing and training.

That's really interesting. And how fast were you able to, like, once you had those basic retrieval evals in place, now, first of all, like, how did you get, how did you kind of construct the golden dataset? Was it like creating synthetic data from their document store, like generating questions that, you know, from documents going in the reverse direction, using an LLM to do that? What was that?

Yeah, it's a good question because there's, there's two parts of that answer. Like, one is a lot of our, uh, FDE, uh, forward deployed engineer and solution architect work is working directly with customers to make sure these things, these things work. And so the short answer to your question, um, is we were, um, like writing a lot of these different tests down trying to do it fast in a loop with the customer and getting quick turnaround time using human labeling. So the— one of the other advantages of a smaller eval set is that you can keep a human eyes on it to be able to see whether you're generally able to, to hit these, these core use cases. And then again, with the thumbs the kind of curated set of thumbs up and thumbs down cases, follow up on that smaller set, um, and, and have a human do the evaluation.

Okay, that makes sense. So you kind of had like a human-labeled kind of dataset that also had allowed you to calculate recall, it sounds like?

That's, that's right, that's right. Uh, recall of the, the, the, um, relevant documents, but then also that the final answer that the, the chat product spit out, um, was, um, a good summarization of those, those documents.

And was there, was there kind of like a process where this thumbs up, thumbs down, you know, you could add— was there like constant adding to the dataset as you got more usage, you know, and you kept curating it? How was, uh, was there a system for that, or—

yeah, there, there was, and it was, uh, you know, honestly a fairly manual process, and it often stays a fairly manual process in companies. We'll talk about another one, uh, Grab, who had a more sophisticated way to kind of farm out these human labels. And that's a, that's a further stage. But really, um, you know, the— until you start having trouble finding the edge cases that are not working, um, I often don't recommend building out spending the time to build out collection processes or making it much more sophisticated until you are nailing these initial cases.

Okay, great. And how so, like in this, like, you know, how much of a, like a, let's say, improvement did you see from doing this eval thing with RAG?

Yeah, let's see. You know, I don't think I, I don't think I have that, you know, specific number off the— well, so there was this metric that I wanted to add here at the bottom here. And this was more of a metric about accessing relevant documents. So I think it touches on your point. So being able to utilize Morgan Stanley's kind of corpus of relevant documents, as you can see here, moved up from 20% to 80%. And this wasn't like— depending on how you're measuring recall across different queries, it's a— this is a little bit different, but it does show that we're able to kind of access more of that full corpus for the, the purposes of, of this tool. So huge jump in terms of an improvement there. And I think ultimately what Morgan Stanley was trying to make sure is that this tool was like meeting a level, a bar that the advisor teams were finding use out of. And so being able to hit, um, something that was useful to these folks at, you know, 98% adoption was ultimately the, the sign of success that, that Morgan Stanley had.

I have a related question. Um, so it sounds like in this case study, retrieval was kind of the thing to fix? I know you have an assistance API that does retrieval for you. You know, that may be newer, maybe newer than this case study. I'm not sure. It is. How does that— does— would the process be similar if you're using this assistance API? Is there anything in that that kind of, you know, helps you facilitate evals? Any trade-offs that you have to keep in mind?

Yeah, good question. So we do, like you mentioned, we have the Assistants API and we have recently launched Responses API, which can also— it acts more like a conversation chat, our previous conversation chat API, and can let you access stored documents in the same manner, just a little bit more easily. So I recommend, you know, continue using Assistants if you love it.

Oh, I may be confused. I get— maybe it's— I was thinking about Responses. Yeah, I get confused between those two.

We have both, and, and, you know, it's a rapidly evolving area, so it's no worries. But, uh, but we do— we did not have that capability at the, the time of this case study. We do have that capability now. Um, and sorry, your question was like how to best utilize it, is that right?

Yeah, I guess like a lot of people may not be familiar with the Responses API. When, when should you use a vector database? When should you think about using Responses API? Is there any trade-offs that you should— while we're talking about retrieval?

Sure. Yeah, I mean, our goal generally with the API is to continue to build out a great platform for folks to use for not just the basic cases of completions, but for a whole platform for agentic behavior. And so the Responses launch recently was a big step forward and it allows the, especially the reasoning models to use tools like looking up information in the files that you've uploaded as well as soon to be launched, being able to use other tools like Python and searching the web. So that's the general direction that our Responsys API is going in. So if you do have a RAG solution that you're looking for, I recommend using the file search capabilities that we have with Responsys. And for that general use case, you're able to use our evals product to make sure that the tool calling process is being utilized at a rate that means you're getting the responses and data back that you were looking for. You can do things like check your responses to make sure that they did a tool call to do a search over your files, and then subsequently look at those results to see if they retrieve the correct files, and there are various settings on the indexing behavior that we have for files that you can tweak in order to improve that. Probably getting into the different ways to tweak and improve that might be out of the scope for the slides that I have, but our long-term goal is to provide that platform where you can continually iterate and get better and better model performance without having to go too far of the ecosystem.

Okay, great. Yeah.

Um, so yeah, big takeaway here. Um, and in terms of like things that, uh, viewers can do in order to, to make the most of this, really it's, it's starting the— if I wanted to demo this tomorrow, what are the 3 things that I would want to show to customers? Or show to users, and then writing down the responses or the output that you would expect the model or the product to give, and then spending a little bit of time reflecting on why those responses are the right ones. Writing down the why part of it starts to build up your criteria, your rubric for, for grading these long term. And again, fine to start with vibes, fine to start with one all in the head of one person, but as you grow this, you'll probably want to be able to utilize some of that written-down documentation. And then additionally, take those— it can even be 3 cases— and integrate them into a CI/CD process so you can be confident that as you change the rest of the system, your core use cases are for sure still working. One way you can get some of these use cases as a quick bonus here, if you're using the platform, as I mentioned, if you use the responses object or completions with, with storage turned on, you will have a log of those responses and completions in your dashboard. You can select those different responses and evaluate them against different models that we have. And as a little bit of a sneak peek roadmap, we are thinking about a lot of ways in which we can continue to automate this process and make eval super quick and easy. So that the platform can recommend improvements for you as you go along.

Can you play that video just a bit more and just want to stop it at a certain point?

Pause it along here. This logs page is enabled in most cases for folks and it stores the completions or the responses that you've done through the API. You're able to click on these different areas and get fuller information about what model was used, the full output, any of the metadata that you attach to it, as well as any tool calls.

If it's a multi-turn conversation, you can see it all rendered inside.

Exactly. As many of your audiences know, multi-turn conversations are modeled. Each request you will be able to see the all the previous turns. So for completions, there is a little bit of work that we're thinking about making easier to kind of grab the most recent one. And for responses, you're able to, to see that history as well. Doing multi-turn evals is a, uh, a, a area that we would like to get better in, but ultimately We think about the multi-turn conversations as a trace of calls and functions that wind up in the result that either your user found useful or not.

You can log everything, log tool calls, retrieval, intermediate steps, whatever.

Exactly. They're all here and searchable. From this screen—

What does distill do? I don't know if I paid attention to that before. It seems interesting.

This is a feature we've been playing with on and off for a while. It's being able to fine-tune a smaller model based on the responses that you've gotten from a bigger model. In this case, we have a 4.0 model. You can select different responses and distill those into, say, a 4.0 mini model. As an easier flow. We're still iterating on that. We've gotten some great feedback from customers that they would like some additional steps in that process to gain confidence about the resulting model. You'll see that distill flow actively being worked on. If you do have feedback on that feature, we'd love to hear.

That's really cool. If you have a curated dataset, so is that doing supervised fine-tuning? The Distill?

Exactly, yes.

Okay.

Yeah, and it is—

And is it meant for it to train a smaller model? Or you could just fine-tune the same model using Distill?

It's meant to, you know, the target and the terminology is definitely the core use cases, being able to train a smaller model that performs as well as the bigger model, just cheaper and faster. But you definitely can use it as a process of using examples that you found correct to improve model performance. In those cases, it's a little bit less about improving model performance maybe and making the model, if you're using the same base model, more reliably return those same answers because you're reinforcing that existing answer that it was able to give.

[FOREIGN] If you're lazy, this is a nice way to Instead of like getting your traces organized and uploading it to, to the other interface, you can kind of like already log it here, and then just like in the same thing, you can just go straight to fine-tuning, it looks like, if you want to. Okay.

Yeah, exactly. And in a similar way, um, this evaluate button takes you through a more sophisticated workflow that lets you pick multiple graders, pick multiple models or even prompts to evaluate with, where this try new models is, is something that we recently launched to make that process, like you said, if you are lazy or just want to see, um, some of the different model behaviors using the same prompt, what those, um, what those outputs.

And what does the try new models do? Does that just like— yeah, what is it?

Yeah, so what it does is it takes the existing completions responses you have and runs that last completion. To your point about the, the multi-turn part, it runs that last completion through a new set of models. So we found this really helpful in being able to just get a quick vibe or gut check of how your existing traffic would perform on a new model. Being able to do that quickly. Now, oftentimes you may want to tweak a prompt to make it more relevant to a new model or take some deeper looks in there, but this gives you a quick way to even scope out and see if that work would be useful.

Cool.

One of the ways we've seen this try new models work for customers actually, I was talking with a customer that had said, oh, I'd really want to try out fine-tuning. I'm pretty excited about it. We want to fine-tune a 4o Mini model, and I just haven't had time to put together my dataset. One of the things I asked, going back to evals, is like, well, have you just tried the 4o Mini model without any fine-tuning? He said, no, not yet. I'm worried about it being not being intelligent enough. Sure enough, he reached out a week or two later and he said, "Actually, I did what you suggested. I just ran it on some of my existing traffic and the 4L mini model works overall. With a few changes, I'm able to use it directly out of the box." That's not going to be true for everybody and it wasn't true for every use case that that customer had, but it did save a ton of time by being able to kind of quickly create these evals from, again, the core places where he wanted to really make sure.

And I think later on this video, if you keep going, when you click the try new models, you're like, it's going to run a bunch of LLM judges. Is that right?

Yeah, that's right.

And where does that come from? Where do the judges like come from exactly, and how are they defined, or do you define them, or is it like, how is it working?

Yeah, we'll touch on this in a little bit, um, but down here you'll see the test criteria, and this can expand. You can add different test criteria if you're developing your own eval, but we call this an autograder, and, and what it is is a, it's a generalized prompt that we've tested across a few different data sets, and it will take a look generally at the the prompt and the response and use a reasoning model to make an educated guess about how good that response was. And I know you've had some, some conversations about this, but we found generally that the models are quite good at grading, even surprisingly in cases where they weren't that good at generating an answer. And I generally like to use the metaphor that It's easier to tell if a cupcake tastes good than it is to bake a great tasting cupcake. Um, and so we found a lot of utility in this. And if you want to take a look at the, the full prompt, um, that is, um, completely available for you in the, the workflow, and you're able to, to change it as well and modify it if you, if you have ideas about improving it.

Yeah, what we tell all our students is like try not to use off-the-shelf stuff. It's an okay starting point. You need to start somewhere and it's good to get the wheels turning of looking at stuff. But what you really want to do is try to measure the LM judge against some human labels just so that you know that you can trust it.

That's right. We make the raw data here available and actually importantly, we did a lot of work in both making sure that the grader itself would, in addition to giving a score or a true or false or pass or fail, it provided reasons why it was grading that way. And we spent— we also spent work on the UX, making sure that those reasons could be exposed through the UI for all of the different examples that went through. Because what we found actually was that when a human would go through it, at first pass, the human would be like, oh, this is fine, like, why did it get a low score? They'd look at the reason, the model had picked up on some aspect of the original prompt or some part of the conversation further up and noticed an inconsistency and maybe it's it's not that bad, but the human would actually pick up on it and say, oh yeah, this is— the model is right here. Like, I need to, to revise my own, uh, judgment. And so my recommendation and, and experience on this has been, yeah, definitely for the long run, making sure that your grader, um, is aligned with human judgment is important, but don't overestimate human graders. They are often, uh, in the long run, surprisingly more fallible than we've seen the LLM as judge graders after they've been aligned.

Yeah, it makes sense. And like, and so this like triple dot thing in the data column, that's where you can download the judge's like reasoning and like its labels and everything?

That's right. I'll try to see here if, um, if this video happens to go all the way into the data. Um, I don't think—

and we don't want to see 100% on every eval, so that, that's not good. That's not good.

Yeah, that's good. That means you're not quite operating at the, uh, at the leading edge of the model here. But, um, but all greens, uh, for, for a demo video here. It was a nice touch. Um, uh, let's see. Yeah, but, um, but I would encourage folks to check out, um, the, the platform, and maybe if we, we have another, uh, session here, I'm happy to, to walk you through some of the details of our particular solution in this area.

Awesome.

So one of the next lessons I wanted to touch on was from Grab, a partner who wanted to go about providing better maps for Southeast Asia. They were finding the maps from existing providers didn't account for all of the nuances in Southeast Asia. Streets that only a scooter or only someone walking could get through. They additionally started mapping out to improve the mobility of delivery and, and, and providing on-demand rides. One of the places that they were using OpenAI for is, given all of this visual data, could the AI recognize street signs, recognize number of lanes, recognize information about the roadway. And they had a more sophisticated, as I mentioned, a more sophisticated eval framework that they had developed, you know, before they started interacting with, with us deeply. And so they kind of started understanding where the gaps both in their own dataset were, but more importantly for us, where the gaps in the model are. And the point I wanted to make sure to drive in here is that—

so sorry, with the mapping exercise, what were they using OpenAI for specifically in the mapping exercise? Like, what— how was that in the loop of that product?

Yeah, so what they would do is, is take the images from folks like, uh, this person on the scooter and create metadata about that section of the road. So how many lanes does it have? What is the speed limit in that area? Is this a a stop sign, traffic light, a yield sign, those aspects about the different road and route, including things like, hey, when we went through this section yesterday, there was a lot of construction. It's normally a 3-lane road, now it's a 1-lane road. You might want to avoid it when doing routing.

Okay. Was it structured data extraction from images?

Yeah. Exactly.

Okay.

Yeah, it's a very, uh, vision, uh, vision-focused, uh, um, need in this case.

Yeah. Okay.

Um, and, um, because they had these strong evals, they were able to both communicate with their own team and drive their own team's roadmap, but also work with us, uh, on the fine-tuning team to drive a roadmap that could improve the vision capabilities of the model. I think that's an important aspect of evals, is that as you scale up the eval capabilities, being able to communicate with companies like OpenAI on where the gaps are, that is the lingua franca across different teams, especially research teams. When you ask the research team for help in a certain area, the first question you get is like, where are the evals?

In like, in this case, was it, was the eval okay? Was it kind of very vision-centric, like bounding boxes on signs, or was it like also like, was it mainly like, hey, this, here's an image, here's a JSON like of like all the different properties of this image, or was it both, or like what was going on exactly?

Um, yeah, so it was, um, they, um, wanted to have an image and just as you said, kind of output structured information about this image, like was there a street sign, what type of street sign was it. Um, and what they found going to this point of, of a challenge is, um, they were especially having challenges with road signs that were partially occluded, or similarly, the number of lanes was hard to judge from kind of the number of cars. You couldn't simply kind of count the number of white lines in the road. And so, yeah, just as you said, and the dataset here is— you can look up in the way in which we do fine-tuning very similar way that you— very similar dataset in which you do evals and fine-tuning.

Okay, you can, you can, uh, fine-tune with the vision-related input as well? I didn't know that. I didn't try that before.

It was actually this company as one of our main alpha customers that helped us develop that fine-tuning vision technology, um, because they were able to say, hey, You know, you're consistently not getting the results from these types of images that we would like to improve. And we knew, and actually this is the point down here, academia or research often has some sort of capability to help you improve model performance somehow. And often it's a matter of prioritizing which of those, um, which of those capabilities and how exactly we want to expose those capabilities to customers that is the bottleneck. And so evals can really, in this case, help teams by pointing out the specific problems that you can look, like literally look at together and understand and then use to see if your technique is actually, one of the things that, that are solving this problem. Um, and so, um, in Grab's case, this was able both to help them focus on the images and eval sets that they wanted to go out and collect, as well as the, the way in which to communicate with, with OpenAI to say, hey, you know, this was better, hey, we wanted to do this type of image. It helped us understand places where we needed to, um, have a difference in the, um, how fine-grained we wanted to process the image as well, because there were cases where kind of wanted to know about the whole image, and there are times when you wanted to zoom in on different parts of the image and understand some aspect of it.

Oh, interesting. So for like, for the zooming part, that feels like it would be pre-API call.

So it's part of the fine— it ended up being part of the fine-tuning process where we could trade off the number of tokens that we would generate from an image. And being able to generate a higher fidelity or higher number of tokens from the image means an increased cost and some risks in terms of, in terms of being overtrained or overly specific in the fine-tuning process, um, versus a lower number of tokens, lower cost, but a better kind of picture of what's going on, uh, overall in the image, a more generalized, um, representation of the image, if you will. And so that's actually a parameter that you could set in the fine-tuning process, but knowing that it was one of the challenges that we needed to provide a lever for. Um, we only were able to learn that because we worked with companies that had evals where we could see like, oh, this isn't working. Um, what about the technology or the process do we need to, to change in order to get it working for, for both use cases?

And so for like— that's really interesting— for these, this like vision fine-tuning, do you kind of do a lot of like similar tricks to like deep learning like data augmentation, like rotate images, zoom, you know, whatever, um, or is like, is that you can just feed it to the API and it does like some magic stuff for you, or how is it, how does it go?

Yeah, for, for, um, for a lot of these vision cases, um, the model has been trained with enough data to be able to handle those those different cases. Um, I guess I would just go back to evals and see, okay, you know, give it a try, find out where the, the edge of capabilities are, and, um, and then we'd be able to, to work with you on, on improving it. And actually, you know, it goes into the next slide, something I want.

So it sounds like from that case you just told me, like, that's kind of like one superpower evals is like, hey, you can kind of mind the difficult cases because the fact that you have evals that you trust and you can kind of, you know, you can curate those really easily and then you can fine-tune on those.

Yep. Yeah, exactly. You can fine-tune, you can iterate on your prompt, and then you can have a, a, a well-known history of like, hey, did I try this thing already and, and did it work? And you're able to kind of get your team aligned and tackling the same exact problem versus, I think I put on another slide here, accidentally shipping your org, right? Like if one team is a prompting team and another team is a data collection team, you might not always be working in tandem and collecting the right data in order to solve one of your main challenges.

And, um, okay, so this story you're telling, you, you know, you talked about two kind of case studies so far, both cases where you really kind of worked hand in hand with the customer. I think a lot of people are going to wonder, I want help, uh, you know. And just to clarify, like, you know, do you have to be spending a minimum amount on OpenAI? Do you have to be like an enterprise There must, you know, I imagine it's not free to get, you know, you have limited, uh, you don't have infinite people.

That's true, we do have to prioritize unfortunately. We like hearing feedback from everyone. I know the whole team is engaged in different conversations on Twitter and other places. Um, we do have a couple ways in which we encourage people to contribute even if it's not a one-on-one Actually, these are some of the, the programs that we've developed where we encourage folks to basically develop evals that are relevant for their domain. And if I can just skip ahead here, we also have some programs in which you can share evals that you've built. And so in your settings page, you can turn on being able to share evals and fine-tuning data, which gets you free evals, sharing traffic, which gets you free tokens, um, and that on, on a per-project level, we try to enable a lot of fine-grained controls here.

So what does it mean to share an eval? Are you sharing like the labels of the eval? You're sharing like— what's— what does that mean?

Yeah, so you're sharing that whole block of data that you see in the data screen. So you're sharing the prompt the response, the grader, the grader response, and any of the, the additional data that you've uploaded. And we use those both training and for evaluating the new models that we would like to launch to make sure that they are meeting the, the needs of the customers that have felt— that have felt comfortable sharing that data, we go about different ways of anonymizing that and, and making sure that it's, it's safe for us to run across.

Okay, so this is like the data that you log through the eval system, like this is just allow you to share it?

Yep. And, and I'll just re-emphasize that, allow you to share it. This is disabled by default, and admins can actually disable the ability to do this entirely. But for, especially for startups that would like to see some traction in solving their types of problems, this is a really effective way of both being able to get some, some free evals and tokens, as well as starting to share some of that data to make the, the models better.

Okay, so there's an incentive if your situation allows, or you've can be comfortable with sharing, then you get some— basically get money in the form of credits. Yeah, okay, makes sense.

Yep, yeah, exactly. And, and we found these to be quite generous credits where, um, companies are, are able to, to, to go pretty far using some of these free tokens. A lot of folks use these for being able to, like, again, specifically for a project in which they're trying to push the edge of the model capability capabilities and provide feedback to us. So that's the general way, but, you know, to your point of how have some of these example custom companies come up, these are companies in which we believe that helping move them forward will help, you know, us bring the benefits of AI to more customers and eventually to all of humanity. And so, um, certainly using the product a lot in an existing way is a good signal for us that if we're able to help you, you'll be able to continue to expand impacting customer lives. So yes, there is a component of, of how much you are using already, but we don't have a hard and fast rule. We're looking for ways and partners of all sizes to, to improve the experience with.

Okay. Yeah, it's like slightly mysterious, but that's okay. I know it's— there's no— there's no, uh, it's hard to have rules. And yeah, it makes sense, like you kind of have to, uh, you know, case-by-case basis.

Yep. Yeah, so rounding out that Grab case, um, again with those evals and being able to, to talk about it in the same way, we were able to kind of co-develop this new fine-tuning Vision fine-tuning methodology, and they're able to improve both their, like, per, um, per-example metrics like lane count accuracy, sign localization, being able to recognize those different types of signs, as well as some more general process of being able to cut operational costs, such as the, the previous process of having humans look at millions of images and hand labeling them, and being able to apply the changes that they see in closer to real time on their maps, meaning that people can get to their places or get their food faster then.

And when you say a speed limit sign localization, I haven't used like, um, you know, this part of the API if it exists. Like, does OpenAI have like draw a bound— to give you bounding boxes around, around things? I haven't used that.

Yeah, so you can use, um, the, the model to ask, you know, as a response, to draw the coordinates of the bounding box around different areas. In, in Grab's case, they are often trying to just pull out the semantic understanding of the sign. So what kind of sign is it? Like, if it's a, um, if it's a traffic, uh, speed limit sign, what is the speed limit at this, um, at this section of road. Um, and as you can imagine, like, OpenAI's corpus of data was pretty skewed toward North American signs. Um, and so being able to better recognize the signs in Southeast Asia across the different parts that Grab serves by using actually relatively few examples the model was able to kind of pick up and start recognizing signs better.

What's the order of magnitude, do you think, for this kind of thing of training data for fine-tuning, roughly, to see a result like that?

It can really depend, and there's a few different, a few different types of fine-tuning that we use. For supervised fine-tuning text, we often— so as a general process, we recommend starting in the hundreds If you're seeing an impact with your evals, continue to scale it up, continue to collect more data. For vision fine-tuning, in this case, they were seeing an impact with literally 100 different examples and then began to scale up from there. With techniques like preference fine-tuning, which gives you a preferred answer and a non-preferred bird answer, the model can extract a lot more information from those two versus examples. And so you often need many fewer examples, in the, in the low hundreds, as a starting point. And then for reinforcement fine-tuning, which we, uh, which is needed for our reasoning models, again, there's a process there where the models are kind of iterating over a single example and learning from a single example and greater. And so again, starting the low hundreds, we often see a great impact from there. Another example of why, you know, having just a few examples of places you want to get better can really impact your performance.

Yeah, great.

And Hamel Husain, I'm wondering if we can pause the recording because I actually have to run to make a—

okay, sorry about that—

my daughter's appointment. No, no, I'm really enjoying this and Yeah, wondering if we could pause it and then, and then pick it back up.

Let me just stop it here.