LLM Eval Tools Compared: Arize Phoenix

So to wrap up this series, the next eval tool that we're going to look at is Arise Phoenix. It is one of my favorite open source eval tools. All right, so welcome back. We, uh, we took a break. Everyone went and took a shower. They're wearing different clothes.

Except for Hamel Husain.

Except for me, uh, but here we are, and we are going to finish this off with Arise, Phoenix Arise. So let's jump right into it.

My name is Sally Ann, and I'm one of the product leaders here at Arise. I'm really excited to walk you through homework assignments 1 and 2 for the AI evals course. Over the next 15 minutes, we're going to cover how to write and test system prompts, generate synthetic data, energy traces, and identify common failure modes using an LLM. Before we dive into the assignments, I wanna quickly introduce you to our two main products at Arize, Phoenix and AX. These two products work together but serve slightly different needs. Phoenix is our open-source, self-hostable, and extremely customizable product. It's designed to meet developers where they're at. So whether you're working in a local Jupyter Notebook or deploying in your own cloud environment, it's great for running evaluations, writing prompts, and digging deep into your application's behavior. And then we have AX, our enterprise platform that supports evaluations at scale. I'm talking billions of records backed by ADB, the Arise data backend with SSO support, advanced observability, and team collaboration features. Whether you're a small team running heavy workloads or a large org with production deployments, AX gives you everything you need to develop, evaluate, and observe our LLM applications at scale. So today I'm gonna be using Phoenix to do homework assignment 1 and 2, but towards the end, we'll talk about how AX could help you complete these with the help of powerful tools like Alex, our AI agent that we've built into the platform. But with that, let's get started. So over here, I've got our code for our first homework pulled up, and we're gonna be focused on system prompt engineering. So the overview of this assignment is to write an effective system prompt, expand our query dataset, and set up Phoenix for testing. The first few cell block here are just for our imports. We're gonna wanna make sure everything's installed and ready to go and doing things like enabling nested async notebook. And then we're gonna wanna get Phoenix up and running. So I'm using Phoenix Local.

It's a lot of imports. I've always noticed that there's a lot of imports, you know.

It's true. I do like that we're seeing this getting kicked off from a notebook though.

Yes, that is a plus.

To complete these assignments, you could opt to use Phoenix Cloud. The only real difference here is, is maybe grabbing your API keys from the platform if you're using cloud. But if you're using local, which I suggest you do, it's really easy to get started and you have a lot of control when you're running the application locally is you can either run this code that I've included in the notebook or go ahead and run this from the terminal. Then once you do that, Phoenix will be running on localhost and we'll be good to go. We have Phoenix set up, great. Now let's do our prompt management. We're going to create a really simple prompt at first. I created something really basic here. I'm just basically telling it what its job is, which is to create recipes and cooking advice and provide it in a clear step-by-step instruction.

Plus points for showing us the prompt.

This is a baseline prompt.

Also, I like she started super naive and simple as opposed to immediately being like, I'm going to think of every possible fuck up.

Yeah.

Yeah, definitely.

And start testing and understand how it's performing. So I'm going to go ahead and log this to the Phoenix platform just by giving it a name, a little bit of description, and I can also log any model parameters that I want, like name. And this is really helpful for when You're maybe iterating over time, you might be experimenting with different models. We're going to be able to track that by each version that we create. I went ahead and ran this and then we pop into the UI. You can see that it's been added. We've got that system prompt along with the model configuration. From here, we're pretty much good to go.

Couple of quick comments on this UI. I find it pretty easy to see what's going on here. You see there's a product template— or sorry, a prompt template right up top. That feels pretty easy to grok. I see there's a place for tools that might come into play later. I feel like model and provider are pretty clear and immediately obvious. One thing I don't see is it's just the system prompt. I assume that if you had other user prompts and assistant prompts, they would just stack in that first top part. Top part. But yeah, I think this is like pretty easy to understand what's going on immediately.

Yeah, it feels nice somehow, the localhost, and you can kind of run it locally without, you know, feeling like there's a lot of magic.

So you'll see as we iterate, you can see all the saved versions and have that full control. Let's go back to the code. Let's create some data before we do our actual prompt iteration piece. Now, the homework assignment comes with a sample of 3 queries. Suggest a quick vegan breakfast recipe. I have chicken and rice, what can I cook? And give me a dessert recipe with chocolate. But the assignment calls for us to expand by adding 10 more queries to this dataset. I added 10 additional queries manually, and then I'm just gonna go ahead and load that into the Phoenix platform. And if you pop over here, we can see that we've got that total of 13 rows available for us. Now, if you wanted to, you could add the examples in the UI and just upload the CSV, kind of complete up to you. I like to do it in code to make it really simple.

Nice tables, you are.

We have a prompt loaded to start with. We've got our sample dataset. Let's go over to the Prompt Playground now and actually do some testing. Let's go ahead and load in that prompt that we've created and our dataset. I'm just going to go ahead and change our prompt variable. I'm using f-string here and I want to make sure that the prompt can actually pick it up. We've got everything all set up. We don't have to do any of this in code. We can just now run the prompt and Phoenix is going to run it on each of the examples. It's also going to track things like latency and token count and cost for us as well. We can see how that compares prompt to prompt as we iterate, but it's also doing an experiment for us. You can see here it says view experiment. It's going to track experiments automatically so you can easily compare outputs across the runs. So it's generating as it's going. I'm going to go ahead and start reviewing some of these results. And let's see how our first basic prompt—

What if I have a million examples in my dataset? Is it just going to run on all million? But anyways, side note.

Yeah, I wonder how— Maybe you can do like a sampling or something.

Yeah, there might be a way to sample.

I like to, for mine, I have them so you have tags and you can subselect down to tags or you can sample over whatever the population that's selected from. One thing that I think is pretty cool and I haven't seen before, they allow you to select what formatting for the templating. So it's like none, mustache, and f-string. That's— I haven't seen that before. That's pretty cool.

Hmm. Yeah, that's pretty cool.

So first question is just a question asking about how we store herbs overall. Looks like it's giving some different ways that a user could—

I can't read that.

Yeah, I mean, let me see. Let me try to zoom.

No, it's, there's no world in which if you zoom, it's gonna be any better. I just cannot read that amount of text.

Hamel, can you increase the font size in the UI? That was a joke. That was a joke.

I know. I know.

Overall, nothing really shouting at me. Now, looking at this example here, it does seem like the format from the first to the second is slightly different. So, um, usually you wanna make sure you have a really consistent user experience. So that might be something that we want to keep track of and iterate. Yeah.

I mean, this kind of goes back to why you want to build your own annotation app or your own data viewer a lot of times, cuz you have domain-specific data. It's not really too domain-specific, even, but it's like, hey, you wanna, you know, read it. Like Shreya's saying, can't read this that easily.

Yep. Also, we talked about this in a previous video. There's no markdown rendering on these. I would, you know, I would love to know if you click that expand. I see there's an expand icon, but I don't know if you were to expand, if you hit the expand icon, if that renders the markdown. That'd be nice. Especially because we know that these models output markdown like crazy.

Yeah. Maybe I should click on it and see.

Yeah, that'd be nice. That'd be nice.

This one too, again, super short. We're seeing such variety in terms of output. The other thing that I'm noticing is there's a little bit of a lack of a personality. The bot's just kind of responding. It's not really asking if there's any kind of follow-ups. And sometimes it feels like it's being either super specific or a little bit more general. So I think I need to give it some personality and I think I need to give it clear instructions on how it should be responding and exactly what styling. It should include. What I'm going to do is I'm actually going to go ahead and push Compare, and what that's going to do is add an additional prompt to our playground. Why this is so powerful is now I can go ahead and take two prompts side-by-side, run them on the same dataset, and be able to compare how they are performing and how they're differing. This is going to be really great for your iteration workflows. I'm going to go ahead and pull in a prompt that I already set up for us here. And pop it into the playground. So let's kind of at a high level understand. So right away, this system prompt is a lot more detailed. It's explaining what the core responsibilities are, outlining what the guidelines are for ingredients and instructions. It's giving it a little bit of a personality and saving what it can and cannot respond to, as well as the style formatting. Those are going to be really key when you're creating— good job reading the prompt— is you're going to want to make sure you're explicitly telling the LLM what to do, what its responsibilities are and how it should respond. And what we'll see as we run this is hopefully now that we have added in all of those additional requirements, we're going to start getting more consistent results and better outputs. So we're seeing the results come in.

I have a slightly spicy take. If you go back just like a few seconds to where you can see the two system prompts next to one another. Yeah, I Every time I built a prompt library, from the very first time to the most recent one I built, every single time, these things in the system prompt are components. They are components that you plug in. And I have always wanted to basically ablate each of them and say, if I ablate the part where I say core responsibilities, what happens? If I ablate ingredient guidelines, what happens? I haven't seen that in any of these demos. And like, maybe I'm like being too obsessive about what's in my prompt and maybe I'm overthinking like the stupid components, but I get really, really like anxious that if I don't kind of one by one ablate these, I don't really understand what's happening in my prompt. So that's just a personal feeling. That all of these have, probably can do cuz they have variables, but I still find myself wondering why are we writing massive system prompts that are just plain text as opposed to components?

That's, that's— Yeah, I think it, it also is in line with the principle of have the least complexity or even to go even further, Occam's razor from, you know, your machine learning mentality. Um, and I think that is a reasonable comment. Like I, I have that same bias as well.

Anyway, that was, that was my soapbox moment. Sorry.

Yeah, no, I think that's a really good observation.

Yeah.

Like requirements, we're gonna start getting more consistent results and better outputs. So we're seeing the results come in. And now let's start doing some comparing. So taking a look at these results, the first ones seem pretty comparable, but you'll definitely notice that a lot less text. So in the beginning I was saying it's very wordy.

I can't notice.

Now I feel like, okay, maybe we're getting a little bit more consistent results there. You can also see right away that the formatting from the first to the second is consistent versus before where it was a little bit inconsistent with how it would present. Even for things like the different, the general questions versus the recipes, you can see that that formatting is really consistent.

You know, so surely, you know, maybe this is just a constraint of the demo, but surely she had to render this somewhere else and do this comparison. I think that's the only thing I can think of that has happened.

Yeah.

Yeah, apparently markdown rendering is like a killer feature. Uh, I'm like obsessed with markdown rendering suddenly.

Yeah, yeah, definitely.

I'm also seeing that it's having a little bit more personality. It's stating why it's picked a recipe or kind of at the beginning. You see here where it says, is renowned for its rich flavors, understanding spices can help your cooking. So I think all in all, we were able to really improve our results by adding in these additional responsibilities and the instructions here.

If you go back just like a few seconds, if you just scrub back a teeny— right here. Okay, so they're showing these statistics. Um, we've got like the time and the number of tokens and the price. Cool, totally fine, reasonable things to look at. But I— this is where I'm very much a like data visualization maxi. I want to see the distribution of those numbers at the top as little numbers or distributions. Like, just show me little histograms of what was the time of each of these? What was the cost of each of these? How many tokens? Because if there's one that's way out of distribution, that's interesting. If there's like a bimodal thing, that's interesting. If I care about these numbers, I only care about them in context. And I don't know what to do with 1,000 tokens as just a point statistic for this one particular prompt response. I don't know if I care or ever will care. But I do care if you tell me that, like, some subset are really long and some subset are really short. Now I started being able to do things with it. So I find myself immediately wanting some aggregation shit. Um, when I build this, I tend to build, like, those aggregation charts on as part of the initial view.

It's going to be really difficult, I suspect, to compete with notebooks. In fact, um, we, Brian, you and I might do a bonus session about—

Just do the whole thing with notebook.

With, um, no tools, which is, we have, we actually have somebody that we recorded that has done that and we can do that if you'd like.

Okay.

I would say one thing further, I would agree, I agree with you, Brian, but I also think that these metrics are like the least important metrics at this stage to look at. Like, do actually read your data. Don't care about the number of tokens at this point.

I agree. I, I certainly am only looking at like token output when I'm starting to worry about certain specific things, not at the moment of like experimenting with the thing. I don't know.

Yeah, yeah. I mean, I think your more general point is that, hey, by putting these statistics on the rows they are more sort of eye candy than they are informative if you were to see an aggregate score.

And we did see that in Braintrust— or sorry, in, uh, LangSmith as well. I don't remember if we saw it in Braintrust, but I definitely saw it in, in LangSmith.

Yeah, to be honest, I don't remember at this point anymore.

Flavors, understanding spices can help your cooking. So I think all in all, we were able to really improve our results by adding in these additional responsibilities and the instructions here. So what I can now do is go ahead and save that. It'll go right as part of that initial version. And if we go back to our prompt over here, what you'll see is we'll now have an additional version available to us.

Big fan of versioning.

All right, so now that we have our prompt, we're ready to move into homework number 2, which focuses on generating additional data and doing more systematic analysis. So we're gonna do this in 3 parts. The first part is gonna be focused on generating synthetic data. Second, we'll run that prompt that we just created on all of those new queries. And then we're actually gonna annotate the results and try to identify where there are maybe failure patterns. We'll then generate our taxonomy for our failure modes so that we can understand what those common failure points are. So step 1, of course, is just gonna be to set up our imports and then just going to get my OpenAI key set up. I'm using that for the synthetic data generation, but you can of course substitute that with any model provider that you'd like. Now let's get into the data generation. So the first step of this is identifying key dimensions. So the assignment asks us to create 3 to 4 key dimensions relevant to the functionality of the recipe bot. Now I'm going with dietary restriction, cuisine type, meal type, and skill level. You can of course adjust these to be whatever it is that you would like. Now This is gonna give us hundreds of possible combinations. You can see here I've created 450, but we're gonna use that to then create unique combinations. And to do that, we're gonna just grab 25 of those 450 and we're gonna create these. So you can see here are some examples as we're just kind of grabbing a few combinations of those. And then once we have that, we're actually ready to do the synthetic data generation. And the way we're gonna do that is actually using Phoenix's lm.generate function.

Why are there emojis in the print statement? I don't know, but we can— we don't need to nitpick on that.

I have a hunch.

And I really want to highlight why using LLM generate—

that I'm a little confused about. My understanding of the exercise was these were the dimensions on which you're going to be evaluating its correctness of output. These are not necessarily dimensions— like, it sounds like she's going to be designing data to span these different dimensions. Is that correct? Is that how I remember?

Yeah. Okay. So the purpose is like, like generate a diverse set of synthetic queries. And one way to do that is to come up with whatever dimensions that you think will have some variance, create queries out of them, and then go do error analysis on the traces generated from those queries.

Okay, cool. Thanks.

In the real world, you probably wouldn't make it as random. You would have some hypotheses and you would say like, oh, like I'm gonna go here and there and I'm gonna like, I think it's gonna break if I do XYZ or ABC. Um, but you know, it is a homework assignment, so I try not, I didn't go that critical. It wasn't that critical of it, I guess, yet.

Is so powerful and it's really because it handles all of the messy orchestration, async calls, retries, rate limits. It's going to work directly with the DataFrame and templates. You just pass it those dimensions in a prompt and it's going to give you the structured results without any extra parsing. It's just going to make this process way easier for us than if we were going to write this all from scratch. I'm going to take those 25 tuples that I've created before, I'm just pretty much formatting it into a DataFrame, and then I've created this prompt for our generation. Then once we do to do that, we can just go ahead and pass that here. And in the end, we're going to get some different query outputs. So we've got our 25 diverse synthetic queries, so we're good to go there. We just got to get them into the platform. So again, we're going to go ahead and, uh, run our upload_dataset call.

I feel like the integrations aren't nice like in LangSmith, you know, where it's just like one decorator, like one line logging, or even in Braintrust. Here it's like kind of generate a Pandas DataFrame of my data and then like run some other function to do the upload. I don't know.

Hmm.

One thing that, I mean, I have used Phoenix quite a lot and one thing I like is it does actually have, I always have Pandas DataFrames cuz I'm so Python-pilled at this point.

You're already in notebooks.

And I'm already in a notebook. And so like, it just works when I look at the docs, it's like, oh, this is what I'm doing already. I'm super biased in that sense.

Back in the UI we go, just to double-check on those datasets. You can see that it uploaded here. Let's test it in the playground. I'm just going to load that dataset into the playground. I'm going to go ahead and choose that prompt that we just created. You'll notice that it automatically pulled in the latest version for us, and all we have to do is run it. I mentioned before, an experiment is going to be automatically created for us. Let's go ahead and take a look at a few of those now. Going ahead and looking at these, what we want to do now is use the free text annotations to form the basis of our error analysis. We want to review each output noting where the prompt didn't meet our expectation. We don't want to simply just say wrong, we want to describe the issue. Like the user asked for a quick recipe, this recipe is requiring something that takes a lot of time. Let's go through a few of these now. This is a great example. So as we're looking through this, this user asked for an advanced vegan lunch that captures the essence of American cuisine. So right away, looking at this, they asked for something vegan. I'm noticing that there's heavy cream, definitely things that are not vegan. So we can do here is type in the notes that say, you know, ignores dietary restrictions. And so, we don't only wanna say, again, that this is wrong. We're trying to explicitly explain what it is that is wrong. So, we can kind of continue to go—

One thing, I have, like, two questions already. So, one is I wonder if you can highlight text and then leave an annotation on the highlighted text. That's been something that I've really appreciated that I use. And the other thing is I don't know what annotation configs are. I'm seeing this, like, no annotation configs for this project. And I find myself wondering, What is this?

Yeah, annotation configs are like if you want some structured, um, you know, fields to do annotation with, like you want a dropdown menu with some values, want a binary value, whatever, you can kind of pre-configure it so it shows up in the form there, essentially.

That's pretty cool.

Yeah. Nice.

Okay.

I think they all have something like that.

I see.

Okay.

Cool.

Through, and what we want to do is essentially continue to try to identify areas where perhaps the application got it wrong. So I've kind of gone through and annotated the full dataset. So let's go back to our code to finish off our assignment. Great. So now that we have all of our data annotated, we can actually go ahead and export that data back into our notebook so we can start creating our taxonomy. I'm just going to go ahead and export that data from the platform. You can see here we get our free text annotation in our results.explanation file.

I really like this.

What I'm going to do is rather than sitting here and combing through all of the explanations and trying to come up with patterns myself, I'm actually going to use an LLM to do it. I'm going to go ahead and create a simple chat completion and I'm giving it a prompt with all of the data and I'm just asking it to come up with some labels that represent the data.

I actually really like the integration with the notebook. It just feels clean and like I can control where I'm at the workflow. I know previously I said I like doing everything in the interface, but maybe I'm having a change of heart.

I don't know.

Uh, you should check out this website called notebooks.club.

Uh, as if I've never seen it before.

Yeah. I mean, you're just reminding yourself of your love for notebooks. Which is understandable.

And so we're going to get is a list of our labels with each data point being a failure mode. And what we can then do is use LM Generate again to label each data point with the corresponding failure mode. And then once we get that, we can kind of analyze to understand exactly where the most common failures are. And we're actually going to save this off for use in a future assignment.

So just to clarify, you know, each vendor you can use it with a notebook and read and write from it. I just think that in Phoenix, that's what they are oriented towards.

Got it.

Like that, like all their examples and that's kind of how they have, they ground, they have like a data science aesthetic to everything. Yeah.

The one comment I'll make is I think Arise is sort of like a, an older LLM Ops company. And so I think they came up a little bit more in the MLOps era. And so maybe that informs why the form factor is a little more like DS.

Yeah.

And so that's gonna be homework assignments 1 and 2. To summarize what we did today is we wrote an effective system prompt. We did that by iterating on the platform. We built a synthetic dataset. And then we were able to create a taxonomy using annotations from reviewing underlying traces. Well, we used Phoenix today. Everything we did can also be done in AX with additional enterprise features. Alex, our AI assistant, can generate prompts and data directly in the UI, and we also have LM as a judge evaluation second run in the playground as you're iterating. So you can get a free account today and start exploring these tools. In Assignment 3, we're actually going to use AX directly to scale these workflows. So this wraps up our walkthrough for Homework 1 and 2. Hopefully this gives you a strong foundation for using Phoenix for prompt-based evaluation and trace analysis. Feel free to reach out to me or to our community if you have any questions, and I'll see you in the next one where we dive into AX. Thanks everyone.

Pretty like to the point.

I learned my note— I learned about my love for notebooks once more.

Back on the bandwagon.

It's always there. You just have to— recall it.

Yeah, I felt like Arise was the most like direct and like it felt, it felt really streamlined. All the steps felt very clear of like how you would go after it. Maybe, maybe now I'm just conditioned by the first two, but I, I don't know. It felt very, very clear on like what the next step in this process was. Less extraneous things going on. I don't know.

Yeah, I mean, it's hard because it's biased pretty well towards data scientists, and we are all data scientists.

Um, well, we are all everything, so yeah.

Okay. Um, you know, but it has some affinity, I think, special affinity towards the data thinking, I think. Um, and, you know, it's a very comfortable workflow that makes sense. And yeah. Um, and then also I have a soft spot for like open source and I like to be able to hack things, hack the backend, the database that is writing to and see it, do SQL queries on it even, you know, like right to, right, right to the SQLite. And then some like, you know, yeah, let's just query it. Um, and do all kinds of things like that.

I will say, I don't know how the production support is going. I definitely don't have data frames running in production. So I wonder how that— I'm sure they thought about it.

Yeah.

Tell me more about that, the production data frame thing.

Oh, like if you're generating real-time traces and you want to log them in real time, the trace is not in a data frame. You're not uploading a data frame. So some other logging util. Probably needs to exist.

Anything can be a DataFrame if you try hard enough.

Okay.

I don't want to make things a DataFrame.

You know what's really funny? I make everything into a DataFrame.

Same.

I don't know. I like everything. It's just the comfort. I have methods on stuff to make it render into a DataFrame. It's like, yeah, so maybe a toxic trait.

Awesome. Okay. So before we go, Shreya, What was the standout for you?

Oh man. I have to say I liked Braintrust's UI. I liked Langsmith's vibe of being product analytics and data and error analysis all in one. And I like the notebook-centric data science-esque aspect from Earthrise.

Awesome.

Awesome. Hamel?

Yeah, there's not a landslide victory here. I think there's something good about each platform that's like unique. I would say like LangSmith has a really strong annotation queue that I really love. Um, you know, I like their playground a lot— how you go straight from trace directly into playground, like right away. You know, of course we're using LangChain, LangGraph or anything like that. Can't beat it. I mean, not going to get better integration, obviously. Um, and there's some other UX things there that are quite good. Um, yeah. And then Braintrust, I really like the UI a lot. Like, it just feels nice. And, and yeah, I just echo, I'm a data scientist, so for data scientists, I like, you know, if I'm coming from a let's say Python data mindset, I like the Phoenix aesthetic.

Cool. Awesome. Well, thanks for doing this with me.

Wait, but what about you though? You can't get away with making a judgment.

What's going on? Yeah, I think I found the UI the most straightforward to understand in Braintrust, but I think my workflows fit easiest into Arise, but I'm not gonna pull any punches. None of these are nearly as useful as the one that I built myself. And not because like it's better than these, just because I built it and it's like exactly what I need for my tasks. All of these feel like I would spend more time learning how to use them than I could just augment what I've already got to be what I need. And I can't imagine, like, I can't imagine investing in any of these until I really was hitting some like production blockers that made it feel like they were gonna be super valuable. Yeah.

And by that time you hit the production blocker, you would have so much information that you would be able to evaluate the tools. You know, you would have some real requirements at that point. You wouldn't be just kind of vaguely like baking them off.

Exactly. Yeah. Awesome. Well, thanks everyone. I really, really appreciate you doing this with me. This is super fun.

Yeah, thank you.

Cool.