Ep. 59: Alex Shan, Judgment Labs CEO

This company was founded on a basis and a mission that you're going to really see self-improving agents one day. The data layer is going to be the people that collect all these trajectories. We really focus on our evals and our analysis portion, which dictates what you measure and therefore what you optimize towards. And so I wouldn't say that you would see us in the future doing a lot of this RL training jobs, but we do provide a lot of the signal lifeblood that those companies need in order to even know what they're optimizing towards.

So let's just go ahead and get started with your background. Maybe tell us a little bit about yourself.

Yeah, so today I serve as the CEO and I'm one of the co-founders of Judgment Labs. We are a startup that provides infrastructure to monitor the quality of agents in production and build self-improvement loops on top of them. Um, you know, as CEO, I deal with a lot of non-technical functions related to recruiting, investment strategy, and product. But I historically have been involved on the more technical side. So for most of my professional career, I was dealing with the engineering and research sides of building agents and NLP systems inside of production. Been doing this for a pretty long time. I got into this while I was a kid, just kind of exploring a lot of interesting open source work going on and getting my fingers dirty in the late 2010s with stuff like NLTK and Stanza. And getting really into all things computational linguistics and NLP. Was very fortunate to attend Stanford and did a lot of research there with my mentor Chris Manning in the Stanford AI Lab. And in the lab, we focused on a lot of foundational data evaluations and optimization frameworks and loops, and was very enamored by this concept of how you can build systems that get better at different kinds of things. And over the course of that development of deep learning into language models into large language models. Got a passion for all things data and evals and how they are related to ultimately how we improve systems almost automatically. And when the agent stuff came around, it was pretty interesting, you know. This is like tool former days inside of research. Got very interested in sort of what this would mean for industry and took a leave from academia and traditional research to go do a lot of applied research out in you know, the real world, worked at Juniper Networks. I was leading one of the teams for agents inside of this, and it was a very interesting, you know, enterprise experience. The CEO, Rami Rahim, brought me in basically from a lot of the prior work that I had done with agents and how they developed and ended up building a lot of autonomous networking agents. I think these are today branded under the company as Mist, but you know, they do a lot of revenue and a lot of great things for their customers. Um, and you know, I was responsible for a lot of the stuff as it comes to evals, monitoring, and self-improvement loops on top of those agents. And so eventually spun that out with a few friends of mine into the company that is today Judgment Labs about a year ago.

Awesome. Talk to us a little bit about kind of your, your co-founders, how you guys met, um, and kind of the journey there. And then we'll get into kind of your time at Stanford.

They're my best friends. So I have two co-founders, Joe and Andrew. Joe is the guy that taught me how to code when we were in fifth grade together in elementary school. And he's just one of these people that's been programming for the better part of his life. And he got really deep into systems and infrastructure. His background is primarily around, you know, monitoring and observability at Datadog. And so that's how he got his start into the game of all this. And then Andrew, who's my other co-founder, I go even further back with. Andrew and I are actually intergenerational childhood friends. Um, that is, our moms are childhood best friends from China. So they grew up on the farms inside of southern China, rice paddy farmers, and they ended up going to high school, college, and grad school together out in New York. And they ended up being such good friends with each other that they wanted to have kids at the same time. So Andrew's kind of my brother. Um, and I was not, I was not having any choice. I was forced to be his best friend, but you know, he is really close with me and he's also the guy that got me into NLP. Um, the history is basically that Andrew's family got me into startups as well as NLP because of his dad. Andrew's dad's a long, long-time person in Silicon Valley who came for a PhD in compiler work at Cornell and ended up writing a lot of really influential work around compiler theory. And then eventually went to go work inside of startups, joined a little baby startup back in the day called Oracle and did pretty well for himself. And then ended up going to Stanford to teach a lot of systems and then now AI there. And so when I was growing up as a kid, his dad was indoctrinating me into the field of NLP. So that's kind of how I got into things. We got together and Andrew comes from a research background as well. He dropped out of Berkeley as one of the earlier members of Together AI's research team where he led a lot of post-training and synthetic data work.

Awesome. So talk to us a little bit about you got to Stanford, what was the year, what was it like, what were the classes you took? And it's particularly remarkable. I mean, you are pretty young and the team is quite young, but you guys are very experienced for your age. Um, how are you kind of able to, you know, do internships and part-time work while being at Stanford? How'd you kind of balance those two?

Stanford was interesting. I came in and, you know, it's a CS-heavy school, so I wanted to immerse myself into that ecosystem. The ecosystems that I ended up going into were very different than what I thought. I thought I would try out software engineering. Eventually went into much more technical roles than just engineering, like highly specialized roles inside of technical teams. I ended up working, you know, part-time, but you know how it is, you know, a lot of people such as myself would just end up working full-time basically. And I joined a startup called Hinkle Protocol, working on a lot of zero-knowledge and identity work on the blockchain. And they needed people to work on cryptography, zero-knowledge, and I have a maths background, so I really wanted to go into that. And I did a lot of work for them. Ended up being like the second hire there under Georgie and Mika, who are the two founders, and ended up doing that, but then got enamored into, you know, continuing my work in AI on the side. So Chris Mann who's, you know, a big figure and a mentor to me and glad to have him as part of our team today as an investor, was the head of AI for Stanford, essentially, especially on the NLP front. You know, there are so many great people, Dan Jurafsky, Percy, you know, but, um, you know, Chris does a lot there. And at the time, he wanted to, you know, extend a bunch of different projects inside of his lab. And so I just started working there during my freshman year and then very quickly realized that AI was going to be pretty big. Specifically NLP was going to be really big, of course. And, you know, this was like 2021, so it was kind of obvious where things were headed already. And so I decided to make the full switch over from cryptography and zero knowledge over to what was happening inside of AI. And so, you know, I worked inside of the academic labs for a bit, published a lot, had a lot of fun. But then I quickly realized, you know, both of my parents are professors and I wasn't very interested in publishing, to be honest. I really liked communicating the work that I had done. I love pedagogy and education, except that I have a real knack for applied work, I think. And so I ended up wanting to extend that into industry. So what I ended up doing was looking for a spot actually in enterprise, because when I was asking Andrew's dad, you know, what's the best way to make a company? I was, you know, consulting Georgie from, from Henko Protocol for advice. The advice was essentially like, look, you should probably study how enterprises move. You should understand how enterprise software is developed, sold, and distributed, um, and marketed. And so I wanted to work in some kind of deep enterprise. And this guy named Rami Rahim, who is the CEO of Juniper Networks at the time, which is this dynastic competitor to Cisco inside of the networking business, was trying to roll out a lot of AI-native software. And truthfully, I think in 2023, which is when I joined in January, I don't think that they knew what that meant. But it was okay because they were still early into networking. And so when I joined, it was sort of a situation where they have some strategies, LLMs are cool, but they want to figure out how to really productionize them. And that's coinciding with a lot of this work around tool calling LLMs. And so I started this little group inside of the CTO's org. Um, and so the CTO was like, look, just rip it. And if it does well, you know, we'll continue backing this and we'll sell it as a part of the core product suite. And so that ended up being very real. And so that's how I got into agent development. And so How did I balance that with Stanford, I guess, is the last part of this question, right? And I think it's just that I was very lucky. You know, I had a lot of friends that helped me get through school, even though I wasn't really doing it much at the time. I'm internally grateful to them. Um, and so it's that Chris himself helped me out a lot. I took a lot of research units, did I say, at Stanford, which, uh, you know, for those of you who are familiar with Stanford's ecosystem, it's a nice way to get credit for schoolwork while doing a lot of stuff out in industry and individual projects.

Awesome.

Yeah.

So now let's kind of fast forward to the founding of Judgment Labs, right? So it's 2025. You kind of have this experience working on multi-agent systems, working on model evaluation with Chris Manning, working at Hinkle, as you kind of mentioned a little bit way back when in 2021 and 2022. And now that you kind of have this experience, you saw kind of a problem and you had kind of a critical insight which was that, you know, there exists this gap for kind of continuously monitoring, testing, and optimizing these agent systems. In particular, when it comes to agent behavior monitoring. And so this is actually like now a very competitive space, but I think maybe when you were kind of seeing it, you saw, you know, kind of a burning problem and a burning desire that you knew must be solved. Talk to us a little bit about why you had such conviction that this was the problem that you wanted to solve. Because I mean, clearly there's so many different things you could do. Multi-agent systems, and in particular, especially with evaluations, um, why, why this specific problem?

Yeah, so I think this was really born out of a combination of the three founders. Again, we're all childhood friends, so even though we were all working at different companies, we talk.

Yeah.

And so, um, you know, it was a combination of Joe's insight on all things observability and monitoring from Datadog.

Yeah.

My time at Juniper for a couple years building agents, and Andrew working on post-training, and You know, these things kind of come together. I also think it comes from a little bit of the research DNA of the team. You know, Andrew and I are both researchers and research engineers, right? It kind of locks it down on the infra side, right? Make sure that product that has legs. But to start with what the problem was, I think it was just that I was building multi-agent systems, trying to deploy them and charge people hundreds of millions of dollars at the end of the day for these things. And it's just really hard to scale those things, right? And of course, like, when you scale those things, there's a lot of things that go into making sure that these systems are reliable and hopefully improving, that you can build a compounding advantage with all the data that you have. But in practice, one of the larger problems was that our evals were really bad at Juniper, and I ended up having to build a lot of the eval monitoring and self-improvement loop infrastructure. And so essentially, in 2023, we started this team. Near the end of 2024 was when I kind of made the decision to step away from Juniper. We also got acquired by HPE for our AI Only have around like $15 billion. So that was a good point. But, you know, at that point, there were a lot of tooling on the market. You had Braintrust who had come out. Indeed. Langsmith had been around for a while. Yep. And I was a practitioner of this stuff and I hated all of them. And I ended up hand rolling a lot of my evals and observability, vibe coded a bunch of stuff at Juniper. And I think that that is a very analogous experience to still what is happening today inside of the markets. And I think that there are like a bajillion things that go wrong with, with agents at scale. But when you think about evals, it's always in search of the improvement loop. And the question for me was trying to understand why it was that my experiences with these products were really not great.

Yeah.

And I think that it started out with the base problems. And just to like horoscope a little bit, I think that for all the AI engineers out there who are building applications on top of these existing piece of monitoring, you might run into problems as it comes to reading all these traces, right? Like for me, a lot of these visibility solutions, as they call them, are really just logging solutions, no different than Datadog or a CloudWatch in and of themselves. So at the end of the day, when you're operating at scale, you know, at Juniper, we had hundreds of thousands and then millions and tens of millions of logs coming through. And there's just not enough people to read these things and really understand what agents are doing. And secondly is that the workflows that were born out of these products were not very native to how I was building agents and how I think other people today build agents, specifically deep agents. And what I mean by deep agents is like we weren't building chatbots at Juniper. We were trying to build fully autonomous networking agents. I think that, you know, we'll probably talk about this later during this interview, but, you know, I think deep agents themselves are also like, you know, probably where things are going inside of agents space for a lot of, you know, white-collar work, but if you are doing that, you don't want to read traces, right? Like each trace could be like 800 steps long. I don't know what I'm reading. I can't scroll through all these tokens. I can't even figure out what's going wrong. I need some true observability in this. I need to get a Slack alert like Sentry when things are going wrong. I need to get some sort of analysis over different patterns of failures across different, you know, things that are going wrong. The way that I conceptualize this for ourselves, I bucketed this into 3 major concepts of observability. You use cases and topics, issues that you face inside of them, and then users that are affected. And so like when you're an AI engineer, usually start with one of these three and you want to find the correlations between these other ones and you get to the bottom of these problems and you solve them and you test for improvement. So to just give you an example, like imagine I'm building an agent, right? If I'm like Claude Code and what I'm going to probably do is someone in my PM team is going to communicate something to me like, an issue-level thing. So for example, we have too many hallucinations. Well, okay, if I'm an AI eng, now it's my job to solve that. And I know that at the end of the day, I'm going to change one of my tunable parameters, right? I have like a prompt, I have a model, I have a context hardness nowadays, yada yada ya. I'm going to change that, but I need to know a little bit more before I start screwing around with our production systems, lest I risk regressing the whole system. And so I'm going to start, say, with an issue like hallucination, and then I'm going to do a bunch of data analysis, right? I'm going to be like, okay, Which users does this affect? I mean, does it affect our most important customers? Because if not, I'm probably not going to prioritize it. Secondly is amongst my hallucinations, what use cases do these arise in? Because ultimately I might have to change a tool, right? Inside of my agent, I may have to change like the description or change what the tool does or change how it works. And so to do that, I would want to understand like, okay, which tool do I change related to hallucinations? Oh, maybe I need to break it up amongst my use cases and topics. So it would be like for networking, it was like, is this hybrid multi-cloud network or is this like single LAN operations that we're trying to do here? And because these are like often totally like open-ended agents where you can tell them to do whatever. So you kind of got to figure this out, you know, to wrap up this point, you know, you have to sort of, you know, take from the other lens, which is what if you have a different issue where it's like, we just want to improve our core workflow. Well, okay, what's the most used feature suite on this agent? Well, it's like wireless LAN, or for us it was actually hybrid and multi-cloud networks. It's like, cool, let me go look at all of my pattern requests for hybrid multi-cloud and then stratify amongst all my most commonly seen issues amongst the most commonly, you know, used customers or our biggest customers. And so that sort of observability was missing to really tie between agent behaviors and issues, the ways that users engage with this, and tying that back to ultimately the tunable parameters that I as an AI engineer have to go and change, right? So that's what we wanted to do with— and that's sort of how we got started. And so there's a lot of stuff that we could go into, but happy to take your lead from here.

Sounds good. Awesome overview. So let's go ahead and move on to JudgeVal, or JudgeyVal, which is this open source post-building layer for agents. And so I think particularly what's exciting about this is you guys essentially open sourced this sort of a community, an agent behavior monitoring software. And you guys did this in conjunction with your own kind of cloud environment as well, which you have kind of the Judgment Cloud. So maybe talk to us a little about, um, I guess like within, you know, Judgment Labs, kind of what are your different offerings? It seems like you have this open-source framework, you have your own kind of cloud where people can run things. How'd you kind of figure out like what your product suite wanted to be?

Yeah, I think the idea around like the open core was really just iteration from the community. I think like, you know, it was really hard during the early days to find people that were building deep agents and would give you a lot of feedback on whether or not your framework was suitable for them, right? So for example, One of the things about deep agents by nature is that their trajectories are much longer and the failure modes become more subtle and they compound really fast and it's really hard to identify where they go wrong. Imagine if I have an even 50-step trajectory on what an agent does and a mistake is made on step 3 and it compounds to step 8 and then down to step 45. It's really hard to find these out unless you have trajectory-level eval, which today, you know, I believe we are one of the only tools on the market, mm-hmm, with their ability that even does trajectory-level eval, and we definitely do it the best out of anyone. And so the reason why we open-cored it was because there are people out there in the community who would both join our team as well as just contribute to the, to basically say, look, like, we think that your eval framework is cool and we like the mission that this is designed for long trajectory, deep agents, but maybe it's suboptimal because of the couple use cases. Like maybe it's just not very good for tool calling evals or something like that. So, you know, that was what Judge Eval was really born out of. The way that we run this company and the way that we monetize is through managed cloud version, right? In the same way that maybe like Databricks is built on top of Spark, right? You know, we're not quite a Spark in the sense that like, you know, we're open core, you know, we're not trying to be the Apache Foundation and build a managed cloud version on top.

Sure.

Really, like, we just manage all of these evals and monitoring in the background so that people don't have to spin up their own cloud versions of this. And so that's kind of how the package has evolved. And, um, you know, I think pretty much everyone that is a paying customer goes onto the cloud version. It's just really complicated to manage all of these evals concurrently, especially because a lot of our customers run up to 80 evals per trajectory. They're trying to extract all this metadata for the purpose of like improving their agents, really. And so to that extent, you know, I think that we'll probably continue to have a mission to open-core a lot of our stuff. We also have a commitment to open-source a lot of our research. So historically, we open-sourced a lot of our research surrounding evals and trajectories. We did a little bit of hallucination detection research back in the day with Chris Manning when I was still at SAIL, and, uh, we'll continue to do that. It's just been a crazy time for us as a company, and so there hasn't been a lot of time for us to write papers.

Awesome. Uh, yeah, so I guess like one of the research blogs that you guys publish is titled like Climbing the Hills That Matter, and, and essentially what you guys do there is you explore some of the challenges with some of the current evaluation methods There are a few kind of key sort of challenges that you guys talk about with respect to evals. One is kind of poor generalization of measurement methods. One is that a lot of these evaluation methods and datasets are being grounded in vibes instead of actual production environments. And then essentially, I think the core insight is that in order to fix it, you need to like track and measure what's happening with your agents and end users in production. And then, you know, kind of create this agent behavior monitoring layer. And so I think there's a lot that we can talk here about kind of the state of evals these days and kind of some of the, you know, problems that they have and kind of how Judgment is able to solve them. But maybe kind of taking a step back, you know, if somebody in the audience is an engineer, researcher that is kind of thinking about, okay, you know, I'm working on evals for whatever end use case, like what would your advice be to them? And I think principally, I think you are quite well, you know, qualified to answer this mostly because you guys have a lot of customers that struggle with evals, which is pretty much why they're using Judgment. That's definitely one of the core reasons. And so you've seen kind of so many different use cases of people that have poor evals. And so I'm sure you must have seen tons of like, you know, like problems. And so maybe you can kind of tell us some of the patterns that you've seen.

Geez, where do we start? First of all, thank you. Appreciate it. We do our best to help our customers. To that extent, the usual way that we operate with our customers is actually pretty high touch. Like I consider ourselves both a product line, but more importantly, we're just a partner in AI reliability to all the people building agents out there. So we have plenty of customers that, you know, use our product. We have some folks that don't even use the product because they're frankly too early for it. And so we just give a lot of prescriptive advice on best practices, like what I'll tell you here. But where do people usually go wrong? I think that first of all, you know, shameless plug, but you should really read Climbing the Hills That Matter. We will continue to push out a lot of these blogs and cover a lot of common failure modes. To reiterate quickly on the two points that's covered there, and then for the purpose of this podcast, I'll assume that people do go and read it. So we'll move on to some other cool ones that we've been discovering lately. Yeah. Number one is that you just cannot use generalized eval methods. This is just one of the things that is an earned insight. And I don't think that you will, you know, talk to a single team that has been able to ship a lot of quality agents in an eval-driven development way without making their evals very specific. And what I mean by that is that a lot of teams rely on these concepts that are supposedly correlated with what you really want to optimize towards, which is like your end user satisfaction rate, your ticket resolution rate, if you're Sierra, etc., right?

Yeah. Yeah.

But the goal of evals is to go from all of our unlabeled production data and we want to get to some optimization over the business cases that matter, right? Like hopefully if I'm a Sierra, I'm like, how do I collect more data and convert that data into some insight that allows me to tune some parameter that allows me to ultimately serve my customers better? And the goal of proper evals then is an eval that both is correlated and measures something that is predictive of end-to-success. Secondly, it's close enough to the parameter layer of what you can change to actually make a difference to your stack. And what I mean by that is, you know, a lot of people use out-of-the-box metrics that are super generalizable and very sweeping and very broad. Think hallucination. They're thinking toxicity, helpfulness, completeness, accuracy. A lot of the time, especially when it comes to deep agents, things are much more subtle when you actually look for the things that are going wrong. These evals at best give you a false sense of confidence into what you're doing. At worst, you're just climbing the wrong hill, i.e., you're not measuring what matters. At the end of the day, the thesis here is you're going to eventually, whether or not you like it or not, optimize towards what you're measuring. If you're not measuring the things that matter, you're done. So, you know, in practice, it's best to be specific. So instead of hallucinations, it's not like hallucinations. It's like, what does a hallucination mean to you? And you might have to get quite granular. It's like, you know, if I'm a customer support agent, it's, you know, historically we see issues with like, you know, refund tickets don't actually go through. Like the API call is made and the agent receives a 404 that it was never made, but the agent ignores it and moves on. That's a hallucination. But if you just prescribe some like random LLM as a judge to say that, it's not actually going to capture what you're looking for. Am I overprescribed? Am I underprescribed? It's just not very specific. Secondly, and is the second most important thing that people usually mess up on, is if you are sort of creating benchmarks and offline testing, it should be grounded inside of the most representative use cases over what people are actually using your apps for, which comes from production, right? So the concept here is like, okay, if we want to optimize for end user satisfaction, How can we devise offline evals that describe end user satisfaction? And that comes in two forms, one of which is criteria selection, which is how do I sort of learn what my users actually care about? And the second is the actual example or dataset collection, which is how do I make sure that my dataset itself is very representative of how people would use this? Both are usually solved by doing really good production monitoring and tying it back and closing the loop of feedback. On the first point of criteria selection, what we do is called rubric discovery. So what we do, especially in, you know, Every agent has its own version of this, but just to give you an idea of this, if I have a conversational agent and a user is saying things like, try that again, here's some extra context so that you can do that. These are all implicit signals that most teams don't mine out of their conversation data and allows them to generate a lot of new criteria and metrics to go and measure. Because if I know that like for a lot of my conversations surrounding some topic, the users tend to ask for some more context and add it to the agent. Clearly the agent is instrumented wrong in some way and I should go and explore that a little bit further. So that's just like a very like contrived example, but there are many interesting ways to look at this. I think a good example to their credit is Factory. Factory made a product called Signals internally, which is really cool. And you know, it shares a lot of the same principles of what we're trying to do, but we're trying to do it on a more horizontal InfoSense and they're making a point solution for themselves. Um, but the second point is where does this data come from for my benchmarks? The data for your benchmarks, it should probably come from production. That's a no-brainer. It represents what people actually use your stuff for. But then secondly, is that it should probably evolve with production trends over time. And this is like one of the things that we point out in that essay is that people usually create a benchmark and they leave it alone for months at a time. And then you drift very naturally as you upgrade different models, your business evolves, your app evolves, people use it in different ways. They run into new problems that are more emergent from new models and new behaviors. And then your eval stack set is still stuck in the olden days. So again, at best you get this false sense of confidence that doesn't correlate to what's happening in production. And at worst, you just have no idea what's going on because your evals are just not representative of anything that you actually care about optimizing towards.

Awesome. Totally makes sense. All right. So maybe to just make things a bit concrete, I mean, there are kind of a few case studies that you guys talk about kind of at a high level without like naming specific names of clients of like, you know, here's a legal case. You're like a Fortune 1000 company that you guys tried with. And so like maybe let's dig deep into kind of like this legal example. Maybe just kind of talk to us at a high level, you know, like maybe what was the customer problem to the extent that you can talk about? Why did you guys use this kind of like jury-based approach in order to solve it? Like, what are kind of maybe some of the key insights and kind of tell us about that journey?

This customer was a really interesting one. It was a very intellectually stimulating problem. And so that's kind of why we put it up. I mean, the results are great, but I think that approach was very innovative and we kind of just wanted to show how there's a lot of work to be done.

Yeah.

Eval methodology standpoint. So basically what happened there was that you have these documents, essentially very representative for most white-collar AI agent flows. Document gets generated by some agent, human expert reviews it and redlines it and then sends it out, right? This is applicable in finance, legal, health, all these sort of expert flows. And for those of you, you know, who are watching this that work on these, maybe this might even be helpful to you. Curious to see how this goes. Um, but If you think about it, that data was not really being used. So like what I mean is like they have, they store that LLM-generated version, they keep the human-edited version, they ship it out. But this is a goldmine of data in terms of evals and rubrics and trying to understand how to improve. Because every single red line that a human expert makes, if you treat it as gold, means that you can correct something in the LLM's behavior, right? So every red line where, especially if you do this at scale and you cluster over patterns of these changes, you notice that the human experts tend to have preferences. Over everything that they're doing. And it's pseudo RL a lot of the time on human preferences. So we take a lot of, you know, stuff that was inspired from there. But the actual use case that we're drawing up is basically how do we regression test and monitor for those changes and sort of understand what the human would do so that we can reduce human review time. Because for these businesses, it's like document gets generated. You know, if you're able to automate a large portion of that human review, your acceleration speeds on like these like agencies essentially is what these businesses are. Becomes so good, right? Like, so instead of having 10 lawyers that get through 100 documents in a week, same 10 lawyers with, you know, a judge reviewer that is calibrated to what they think is going to get through thousands. And so essentially what you have here is a system that was calibrated on these like natural diffs inside of these preferences that the humans are explaining. And, you know, today we have even better systems of this where you can basically run a bunch of LLMs over it to reason out over clusters and do even more interesting clustering work. That was like, you know, this approach that we did in the case study is literally months old. And so, you know, it's a very interesting, interesting study in how you can use all these implicit signals. And I think that like to, to their credit, a lot of the model labs have also realized that they shouldn't be doing this. So Claude Code, for example, who has a great team behind them, is doing a similar thing where it's like, hey, you know, you have code that is suggested by an AI. They can log how humans go and edit that code and what actually gets merged. Secondly is that they can also correlate between human preferences and patterns on things that are accepted versus declined and really like generate out real rubrics over both user and agent behavior for, you know, not only what is preferred, but also what agent behaviors led there. And that's the interesting part. If you're an agent developer, it's like, yes, yes, yes. Like the product people on your team are going to be like, look, we just want less hallucinations. And you're just like, yeah, you know, no duh. You know, I want less hallucinations too. But as an engineer, the question is, well, you know, is it this tool pattern that correlates with like high success over these kinds of workflows? Or is it that a singular tool seems to be, you know, really highly correlated with a lot of these failed cases in which users aren't happy? So that's, you know, example of what we did for them. And yeah, we've continued to keep the ball rolling for a lot of our new customers.

Awesome. Yeah. So I think one other thing that's really exciting, and I think we already kind of hit on this, is like this open source community that you guys built, but I mean, particularly there are a few different use cases that you guys kind of highlight. You guys have a repository that's kind of like a Judgment Cookbook where basically you guys describe maybe like a few different cases of how you would maybe use Judgment, kind of like JudgeBell, this package that you guys had. And one of them is RL, one is online monitoring, one is custom scores, one's offline testing. So maybe just talk to us a little bit about kind of the different applications that you guys are seeing where people are actually using agents. Like obviously there's a ton of work going on with training agents with RL these days, obviously monitoring them, building custom evaluators. Like if you had to kind of like map out the ecosystem right now of like where you're seeing most people use it, like what would you kind of say are kind of the main applications you're seeing?

Primarily all in monitoring. I think that like, you know, evals is a really big place. Yeah, I think that most people think that evals is really crowded. I think that's just because we don't have good taxonomy to separate what the different players are really good at. Because evals is really just the study of like, you know, the limitations and capabilities of your system, which, you know, no crap, is going to be a really big business for AI.

Right.

And so, you know, there are big businesses that are going to be created inside of offline testing and regression testing in a fashion, a new form of CI for agents. Right. There's also going to be businesses, you know, hopefully like ours that focus on CD and like what happens inside of deployment as to how customers are affected. And of course, all these players will try to converge and consolidate. And so the entire agent development lifecycle is stored inside of one company. But for now, you know, we center on incident monitoring. And the reason why we do that is because we think that if we control the production layer of what's happening with agents and we monitor that, that ultimately cascades into everything else as to how you do RL and how you do offline testing and where you curate the datasets and the metrics. And specifically, you know, one thing that we haven't covered much today, which I'll just touch upon, is like, what is the possibility for RL here? I think that, you know, I'll be honest with what we're great at and what we're not great at. I don't think that we have the best RL team in the world. That probably belongs to people like Thinking and Applied Computing, these kind of companies who really know what they're doing with training these models. What we will always do is control an infrastructural layer that dictates what those companies optimize towards. This company was founded on a basis and a mission that you're going to really see self-improving agents one day, and self-improving agents are going to operate on 3 modes of infrastructure. Evaluations and analysis and optimization. The data layer is going to be the people that collect all these trajectories. You know, today a lot of these logging solutions exist today. We provide logging and tracing too. We really focus on our evals and our analysis portion, which dictates what you measure and therefore what you optimize towards. And if I can combine, you know, in the abstract, what did my agent do, as well as whether or not it did it well or did it not well and where, right? That gives me perfect primitives to line up to go and do optimization flows because I have everything that I need to plug and chug. And so, you know, I wouldn't say that you would see us in the future, you know, doing a lot of this RL training jobs, but we do provide a lot of the signal lifeblood that those companies need in order to even know what they're optimizing towards.

Awesome. All right, so let's maybe like take a step deeper into this. So, um, there is a lot of competition in the agent kind of behavior monitoring space. There's another startup called Raindrop that actually has some like really amazing clients as well. and I think they're also doing some really exciting work. And then obviously, you know, the, the larger incumbents just generally when it comes to observability that definitely like understand that agent observability is an important thing to deal with include like Datadog, for instance, where, you know, one of your co-founders also, you know, used to spend some time there. So talk to us a little bit about, you know, from a competitive landscape, you know, as CEO is also, you know, something that, that you spend a lot of time thinking about, which is like, um, a lot of this has to do with building relationships with customers, making sure that they value you, obviously. You guys spend an exceptional amount of time like really getting to know these customers and focusing not just on building a platform, but actually like the deployment of, you know, this platform with customers and kind of ensuring like successful deployment. But like, so is Raindrop and like, so is, I would assume Datadog and all these other places. And so when you kind of think about Judgment as a company, why do you kind of like, what are, what are like the specific areas that you feel like you spike in or the company spikes in that you're confident? I think we've already covered a few of these, but like, what are, like, if you had to summarize, like, what would you say the main ones are?

You know, I aspire to be as good at design and UX oftentimes as like a lot of the people in this space. I think, you know, we have a great appreciation for the guys at Raindrop.

Yeah.

You know, really put together a really nice UI and they've figured out design really well.

Yeah.

And probably learn a lot from them. You know, I wouldn't say that our platform is poorly designed.

Yeah.

You know, you can probably look at LangSmith if you want to see a really poorly designed product. Yeah. I think that if you want to look at, you know, design, they're, you know, top class. I think that where companies would probably look to us to find differential value is on the two fronts of how deployments are done to customers and how research is done, what that will mean for product in the long run. So, you know, I have a core belief that we want to be the evals-first platform for agent behavior monitoring. And what I mean by that is that You know, again, going back to my example, yeah, you know, my mental map of you have data, you have evals and analysis, and you have optimization. I would always prefer to be in the first two. You know, I love questions around data and evals and all these things. And so we will be that kind of company. And I think that we put together probably one of the best teams in the world surrounding evals. And what that ultimately means is that we have a lot of AI talent and like really, really deep research talent as it comes to data quality and eval. And what that is in terms of the product is that we often come up with methods to do things much more accurately and scalably on top of these systems. You know, I think we might be literally one of the only companies in the world that can monitor every single agent call that comes out of these companies that we serve and apply over 80 evals to them and guarantee boundaries on the accuracy of these automated evals. One of the problems that a lot of people come to with other evals companies or agent behavior monitoring companies is that the signals that they capture and the evals that they capture usually suffer from two problems today. One is that they're just not very accurate. And you know, it's a really difficult problem to solve when you don't have an AI-heavy team and you're not evals-focused. Secondly is that they don't work on deep agents. And what I mean by not working on deep agents is like, the question is, can you run an eval on something that exceeds the context window of an LLM judge? Because a lot of these companies, what they do under the hood, if I'm being real, is they take a JSON of your trace or your log and they throw into an LMS judge and they just say, what's this? Of course. Here's the rubric, right?

Of course.

We do it a little bit differently. I won't spill all the secret sauce, but we can handle insanely large context because we do evals in a completely different way that allows us to, A, manage larger context, which means that even if your trace is like 10 million tokens, we can eval it up to 95% accuracy for most of our customers. And secondly is that we do it cheaper, right? So like, it's not even an argument of like possibility, because like right now there's literally no platform in the world that does this. So if you were to chuck in a 10 million long trace into any company, because they all do the same thing with it, they chuck it into an LLM as a judge. If it exceeds the context window, their systems break. They just refuse to produce results. And then secondly is that in reality, as most LLM developers know, the results past 100K tokens even are abysmal. Context rot is a very real thing. And so when you're looking for very precise signals, like, you know, a certain tool leading to some downstream error, when you don't have really good evals infrastructure to handle that, you know, your evals are kind of pointless in the first place. So I would say it's that. And then the second part is just like customer deployments. We are extremely high touch and this is all purpose. We're still early as a company, you know, we've been around for a year and I'm very grateful that that year has been filled with hypergrowth. And I think that's a testament to the understanding of the space that our team has. But we are completely obsessed with serving our customers, meaning that almost everyone that we work with, you know, I think that the, uh, the various teams on, on our website can attest to this is that, you know, we respond to queries in less than around 2.5 minutes on average on Slack. And we hop on calls to do deployments for people because I really care about people's agent systems getting to self-improvement. And we will build the evals for people ourselves and we will give them the best practices and we will leak all of our research alpha to make sure that our customers are really happy with their, you know, systems that self-improve and get really good evals. And I think that that high touchpoint means that I can't serve as many customers today as a lot of these other companies because we're not necessarily designed to be a self-serve product today, if I'm being honest. We will be there. Hopefully we'll do a self-serve once this year. And you can't self-serve a lot of our stuff.

Yeah.

But when it comes to evals, we take a lot of pride in doing it right and not just putting out some, you know, flimsy LM as a judge that gives you a false sense of confidence.

Totally makes sense. I think there is something to be said, and I think this is also a core value of Delta, is quality over quantity. And I think it can be very, very tempting to build a self-serve system that will get you more quantity of customers, but then it's about the quality of what those customers have to say about you. And I think there is something around having a small community of people that are truly incredible or incredibly high quality, but also a small set of relationships that are incredibly high quality. And so, I think, I mean, part of the reason why we also wanted to have on the podcast is because you guys have this focus on quality of relationships with your customers that is actually very out of distribution in the Valley right now, especially when it comes to the whole AI boom. And so like we actually really do respect and value that. All right. So maybe kind of moving on here to maybe one of our last questions. So, you know, you have obviously had a really, really amazing, you know, sort of trajectory, you know, doing this. No pun intended. With, with your work at Judgment. And I, and I also like feel like it is very impactful and it is very important. The field is changing super fast. I mean, there's always constant, you know, new research coming out, new things coming out. Now there's a whole, you know, new buzzword of continual learning that everyone's also thinking about these days. And I think what's interesting about Judgment is it's not really a pure software company in the sense where there's no focus on research because as kind of the bar shifts for evals, like a lot of this actually is directly relevant to some of the work you guys are doing. So you guys are kind of in this, this unique position where you have kind of a foot in the research door, constantly being aware of all these new evals that are coming out and exciting things coming out, but also kind of a foot in production where you're, you're essentially like trying to take this research and actually move it to real companies and produce real value. And so now let's talk a little bit about like, you know, you guys have clearly seen so many different problems, you know, by building these, you know, really authentic relationships with customers in the deployment side. But you guys are also, you know, constantly either attending conferences or, you know, kind of, you know, talking to researchers and constantly aware of kind of the future of what research looks like within evals. And so, um, you know, we had Kartik Narasimhan on the podcast, for instance, who, you know, used to be head of research at Sierra, worked on, um, you know, a lot of really amazing things, including TauBench. Um, you know, we had an event earlier with one of the people that worked on TerminalBench, for instance. Um, we had, you know, the SuiBench folk, um, uh, you know, John Young also on the podcast as well. And so, um, you know, I think maybe the viewership, the audience is relatively familiar with a lot of these different sort of like benchmarks and eval techniques. Where do you kind of see the future of the field and where— how do you kind of see that future tying into the work you're doing at Judgment?

It's a very good point. I think that the DNA of this company is a nice blend of research and engineering, and you can see it in the founders. Yeah. Andrew was training at Together and I did a lot of applied research and before that just all evals and benchmarking work. We have a deep appreciation for the depth of which you can go inside of evals and make that to the general public inside of AI. Evals, all they know is these like boring-ass benchmarks with these like deterministic often metrics. And these things even mean anything is the question. But then like it's very barbell because then you see like the best teams in the world, especially at Anthropic, that build internal agents are all evals and all they think about all day is better evals. And so, you know, we tend to surround ourselves with the people on the, the right end of the barbell, if that makes sense, to really discuss with them. And we share all of our research with people at Anthropic and OpenAI because we just want them to understand that they need to build their systems in preparation for these things. And they help us think through them as well. And the concept here is basically that evals is a lot more deep than most people initially think. For example, like most people think that eval is just chucking stuff into an LM as a judge. And today you're seeing people get burned by that. You have a lot of posts like, hey, like my LM as a judge fucking sucks. You know, it's like, of course, you know, we would push the boundary on that and help people build more reliable eval systems in and of themselves, especially when they're automated.

Yeah.

And also, if you think about like, where is the future lying inside of evals, you know, it's very hard to say, but I'll give you perhaps a perspective that is a bit more interesting. If you go back to the characterization of data evals optimization in this endless loop, and this is how you create continual learning and flywheel systems. I agree. I agree. I think this is how continual learning is going to work from an infrastructural layer. But one of the things that we're rolling out quite shortly is called Data Intelligence. Data Intelligence is basically coming from the observation that these 3 infrastructural blocks are much more blended than most people would think. And the why is because your data itself, even though it comes in unlabeled, it's traces, it's logs, it's unstructured, you know, what are these things? You can enrich them to be eval ready and tell you a better story about what your agents are doing. Historically, The entire observability space is predicated on this idea that, you know, these are separate concepts. The logs are separate from the evals, which are separate from the optimizers. And I just don't think that's true. You know, one of the things that we've been finding is that you can enrich your data that comes through traces to be more eval-ready and more high signal for then an agent judge to go and examine and find interesting patterns out of. You can reformat things in the ways that traces are structured that enabled, you know, agent as judges to examine them and find these, you know, insights so much faster and more uniquely. And you can do so much interesting work along those dimensions. And so to that extent, one sneak peek into what we're doing is this concept that we're going to merge the data and the evals layer together from a production monitoring sense. And I hope too that one day we'll find interesting analogs to that in optimization. Don't know, right? Like an easy way to see this is that like, you know, beforehand people didn't really know that rubrics, which are correlated to evals, are exactly what you need to do RL. A lot of the time, right? And so like the optimization layer is already starting to see that rubric discovery is like one of the most important things to find. So there's a lot of fresh research on that that's coming out in the last few weeks. So, you know, I guess this is all to say, Ankit, I don't know, you know, a lot of smarter people than me are working on stuff in evals. We think that the data layer is very interesting to work on because even if like all these systems get better and who even knows what happens to these agents over time, if cloud code just takes over what to do. But at the production app layer, you still need to know what's going on. You still need to know what's happening inside of your data. You still need to know where your systems fail and how to steer them towards getting better. So that's pretty much what we believe to be core.

Awesome. All right. Well, on that note, thank you so much for coming on the podcast.

It was a lot of fun, man. Thanks.