How Engineers and PMs should collaborate on Evals

Session is about how engineers and PMs can collaborate. Aman is an AI product manager at Arise. He can describe what AI product manager is. We kind of had a lesson earlier that like went deep into that, but we can briefly go over it. You know, there's a lot of questions involved, like who does the evals, who's driving the evals, particularly like who's writing, who's Who's like labeling things when you're part of a company? People like to point fingers like, no, I'm, you know, it's like, like just pass around the labeling as a hot potato. No one wants to do it. So who's doing it? Is it the PM? Can you give the labeling task to your PM? These are, these are interesting questions. And, uh, Aman and, and I are gonna try to come to an answer.

Awesome. Yeah, this is super exciting and good to see you again, Hamel Husain, and, and a lot of folks here definitely see some familiar faces. Cool. I can do a really quick sort of, you know, maybe it's helpful just to do a bit of level setting on the PM role in this AI, LLM, GenAI sort of world a little bit. And then we can hop in from there, Hamel. Feel free to ask any questions. And we have some good debate, I think, around this. So, uh, so for the folks that, you know, that, um, that I might be new to, um, my name is Iman. I lead up our product team at Arise. We're an agent eval, agent and AI application eval and observability company. So, you know, we actually work really closely with Hamel Husain. He gives us really helpful and useful feedback on our product all the time. So we really, really appreciate like you know, people that are sort of getting their hands dirty and actually trying to build improvement into their, their workflows. Uh, and just by way of background, I started my career working on self-driving car evals a number of years ago and then worked on Spotify's machine learning platform. So I've been in the space for a little bit of time building and shipping AI products. Um, so, so pretty, pretty familiar on both the, the infrastructure side as well as building end products for users. I thought it'd be kind of fun just to make this like less of a slide show presentation and more of a whiteboarding sort of session. And so we can sort of like think of this as like marking up a whiteboard live. And if you guys have questions as well, just feel free to drop them in. I'm happy to try to kind of take them live as well. Hamel Husain will sort of be making sure I'm asking— I'm going the right direction here. You can see him in the whiteboard too. So This is like a very quick just brain dump of like when you hear the term AI PM, this is kind of what I think about. There's sort of 3 flavors of, of AI product management. There's the core product itself where AI is a part of the product. Um, and that's really where a PM is thinking about like, what should the form factor be? How do we package this technology up? What should the end product look like for a user? Uh, there's sort of platform builders and this is you know, within an organization or even it can be overlapping with the product person and product person, but they're thinking about infra, managing costs, managing reliability, you know, do we have the right model? Are we picking the right model for the application? And then there's the, the AI-powered PM. Again, this is like, there's, it's not really mutually exclusive. I, I sort of view these as like almost, you know, sort of overlapping Venn diagram of someone who's using AI in their job. And this is where a lot of the like prompting guide, you know, prototyping with AI, uh, maybe even writing code sort of comes into the picture. I bring this up as sort of like a bit of a reflection on the PM role. And many people have worked with product managers in the past. And, um, even me just being in the role, I feel like the expectation has really changed on what product management means. Like back in the old days a little bit, this was like the diagram people would show. It's like, Product looks like this, and then engineering does this, and design does this, and then sales and marketing are off doing their own thing. And now I feel like we're living in this world where these roles and responsibilities are more and more overlapping every single day. And instead of thinking about roles and responsibilities, I sort of, sort of try to push people to think about like skills, like what is your skill? What's your niche? The area you really care about, the area you're really good at. And if it happens to be product. You're not going to get away with, you know, just being able to do your own thing. You have to be able to collaborate with AI engineers and sort of vice versa. If you're an engineer thinking about how to get the most out of your AI product manager, that's really what I'm hoping to impart from this session a little bit. So that's sort of just to set the tone, the context of like where product management fits into this new AI development lifecycle.

I put a funny little— I made a little funny edit to it. Do you think, uh, you you know, this is there, is it like we've seen this diagram before in like data science, you know, there's like statisticians and software engineers and, you know, I forgot what the third one is. And then in the middle was like this unicorn data scientist. Is there like, are you trying to say like, hey, we're not, you're not really saying there's a unicorn. It's like, it depends like based on like your other graphics. You know, you have to like kind of choose where your strengths are.

Sort of. I, another way to look at it is like, I'll probably come back to you in a year, Hamel Husain Husain Husain, and draw this diagram like this again and be like, actually what we really want is like a full stack AI product manager that can kind of do all of those things. And then if you were spiky in an area, great, that's your strength. But I don't think you're really going to get away without being a little bit of like thinking about all of these things to some degree as a PM. So I'll give you like another example. We were just at, I was at AI-Eng World Fair last week and it kept coming up. Like I gave this whole talk on like evals for AI PMs and the Q&A 30-minute section at the end was like, you know, everyone really latched onto the point where I talked about like how our PMs contribute code back. And a lot of people are like, wait, you guys write code? You guys write labels? You're writing evals? So I think there's a lot of, a lot of folks that are getting sort of you know, more comfortable with the fact that product management is going to look a little bit different now that, um, the barrier for creativity is sort of like, you know, actually creating something has gone down.

Um, makes sense.

Yeah. But good point. I like that. I'm going to keep that in there.

Um, great.

Awesome. Um, so I think maybe just going on a little bit in this journey, um, I thought we could just talk about like what things kind of look like today. Like, I really liked the— I took a look at the course reader and I love that like vibe coding is pulled up in there. And I thought I would kind of maybe just express the three phases of building AI products that we often see. So for context, like, from where we sit as a company, and if you haven't, if you haven't heard of us, um, I'll just like briefly show the like landing page just so that there's some context. Like we work with some of the leading companies out there that are building AI products like Reddit, Roblox, Booking.com, largest travel site in the world. And what we often see is there's like a spectrum of where people come to us and say, hey, we should maybe start thinking, should we start thinking about evals is usually like a very common question. And, uh, One just sort of example workflow is like, I would call this like level 0, level 1, like you're a startup or maybe you're building something internally for your users and you just want to move fast, right? The vibe coding workflow. Maybe you're starting with like building an agent or AI product. Like we can build something over the weekend now with CrewAI or LangGraph. You take a look at the outputs, you kind of do the like, looks good to me. And then you hit ship. And what ends up happening is like, you know, usually that product actually does a pretty decent job for the most part for like some of your users, but then you're going to have some users that complain, right? And this is like the internal— this thing of this is like an internal tool or code that's a little bit more, um, you know, it doesn't necessarily have to have staying power, right? You're okay with shipping something that's not great. This type of workflow. Works if you, at the end of, of shipping something, you're not worried about like damaging your brand reputation or you're not worried about, you know, regulators coming after you because you accidentally leaked some customer data. It really isn't a realistic workflow for enterprise, um, you know, sort of product builders, but I want to throw this up as like a lot of people who are vibe coding, building stuff with like Replit v0. Shipping something quickly, like the, the thumbs up, thumbs down vibe, checking the outputs. Like, I think that's great for like internal tools. So there's a certain category of tools where this makes sense. I think the fact that you're taking a course on evals means that you're probably further along on this workflow and you might be thinking about where evals sit in your product stack. And in this case, when you're, when you're building the, the agent or the AI product, and you're writing an eval, a very common question that comes up is, do we trust the eval? And, uh, we get asked a lot, like, do you guys have off-the-shelf evaluations for, like, toxicity or hallucination? And if you give us that eval, should we be able to trust the output, uh, from this, this eval system? And that's usually where I think things start to get pretty interesting because that's what involves, you know, kind of forcing you to like look at your data, look at the outputs a little bit more. So I kind of call this like the, okay, we wrote an eval, we tested something, but we're still at this point where we're like, do we trust the eval? And this is really where Hamel Husain and I have like, have had a ton of debate, which is who writes the evals and who writes the labels that you're using to actually check those evals. So Hamel, I'll pause there. I don't know if you want to add anything else more here, but we can go a little bit deeper on this. Oh, you're muted, by the way.

Um, so I think the undercurrent is everybody wants to pawn the labeling on someone else. Like, if we were to announce today or declare that PMs should write the labels, then everyone would be like, hey, like, I'm not writing the labels, I'm gonna give it to my PM. And that might kind of result in some sort of backlash. From PMs and they might say, hey, like, I'm not the person necessarily gonna write the labels. Like, I don't know how they would feel about that. Um, it's, it's interesting. And so I think this, like, historically labeling, even like personally, I love labeling and looking at data. I think maybe the students in this course will appreciate the value of that. But I think in companies and organizations, like, people are really busy for whatever reason. That labeling tends to, yeah, it's like minimized in importance somehow.

Yeah. Could I get a quick like show of, I know this is like a virtual one, but maybe we can do a quick show of hands. Like, could you drop in the chat if you've actually like labeled data for an eval? Like how many people here are just like rough? You can say like, yes, me or whatever, but I just wanna kind of see like what percentage of your, your Audience is like, okay, cool. I have and I'm the PM. Yes. Amazing. Okay. So, nope. Make the scientist do it. I think that that's like kind of what I was curious is like how many people have written labels versus how many people have pawned it off and like what that ratio looks like. And I, I guess what I'm getting at is like these two workflows are actually pretty interlinked. Like I threw in PM here as like maybe someone who should be writing a label. But to be honest, like these two things are a little bit one in the same to me in that you can't really like the label that you're writing should impact the eval, right? Because at the end of the day, if you're a PM, you are responsible for the end product and you're responsible for that end user experience. And so when you think about an eval system, that contains an eval prompt, an LLM, and some data, you're ultimately trying to say, okay, maybe I can tweak some knobs on my LLM. Maybe I can pick a better LLM, you know, cost performance, latency, accuracy tuning. But the data that you're using to check that against that eval LLM, maybe that's a dataset of like input output, expected label, expected output and label. This is, this is really, really important, right? Because this ultimately determines what the goodness of your overall system is going to be compared to like, you know, like what that end product experience will look like for your user. And I think similarly, the eval prompt, which is defining what should be evaluated, that I feel like is also a collaboration between your PM and AI engineering because your eval prompt should be representative of your data. Um, I'll kind of show this, this graphic here a little bit of like what an eval prompt sort of looks like, just to kind of like, you know, I know, I know everyone here is probably familiar with this, but when you think about setting the role, providing context, providing the goal, and defining the terminology and the label, these are all like natural language. And you have to usually provide examples that fit into the eval prompt as well. So deciding what examples, what few-shot examples go in, defining what that label look like, looks like. You can't really do that unless you're labeling the data and you're in the data itself. So I would actually push back a little bit, Hamel Husain, and say like, you know, as much as like the PM is the right person to write the label, they're also probably the right person to kind of help write the eval here to some degree. And so I think that there's like this, this good kind of healthy tension of like, maybe it's not as mutually exclusive and it's a little bit more collaborative in terms of a workflow. Where, you know, the PM takes on some of the labels and then the, the AI eng kind of can help iterate on those.

I would like, I would like to see the PM. I think the closer you can get the labeling to the domain expert, the better. And usually the PM is more of a domain expert than anyone else, if things are going correctly and organized correctly.

Right. That's, that's exactly right. And then, and then at the same time, if the AI engineer is the person accountable for the end product. Product delivery, then the eval prompt and the final prompt that goes into the system also like kind of rests on the AI engineer, but it also is a little bit of like some domain expertise should enter that as well. So like you should see, you know, a little bit of that like back and forth, I think, in a typical workflow. We see this a lot. We see human— we see PMs writing labels. We also see PMs writing eval prompts and writing the end prompts, um, quite often. Okay. What I want to kind of— where this kind of leads me is like, I'm not sure if we're going to get a packaged up answer of who writes the labels and who writes the prompts necessarily, though I do think that like leaning towards PM should write labels is probably a fair assessment. Like the PM needs to be in the details of the data. The PM needs to be in the details of the eval, but kind of who does what and what that split looks like. Your mileage may vary at your company. What I would urge people to do is think about these 3 questions, which is when you go back and you leave this course and you move on to like writing and building AI products, I kind of ask people, how many labels do you think are good enough? And for each example, how many labels should you have? And so this is a really important question because like a lot of times people will come to us and say like, do we do, you know, is 10 labels enough? 100 labels? 1,000 labels? And I think that this is a question that PMs should be working with their AI engineers on, which is what is statistically significant for you to know that this eval is good enough? And what, you know, does that matter per row? Should you have agreement between all of your subject matter experts, or are you okay with disagreement in your human labels? So think of this as like, what's the quality and quantity of the human labels that we have? The second is what happens when the eval says that the output is good, but the human label disagrees. And what happens during this tiebreak scenario? And you'll kind of see this when you actually get your, your, your product into production. It's going to happen a lot where your eval system is going to say this went well, but your human in the loop or your feedback from your end user is going to say this was a bad response. So who takes a look at those examples and what happens and who's the tiebreaker there? So that's a, that's a really important question that comes down to like your team. And then I'd say last but not least, what happens when you—

we can linger on that for a second. Yeah. One of the pieces of advice we have in our course reader and otherwise is wherever you can try to anoint a benevolent dictator. Mm-hmm. And, um, you know, that benevolent dictator is someone again who has the most context and the most domain expertise ideally. When it comes to doing the tiebreak. Maybe you would be an advocate of the product manager being the benevolent dictator.

I think it's, I think it's important to look, I think, uh, there should be a process in defining who that person is. I think that's what I'm advocating for here, which is like, whether it's democratic or not, I don't know what the process looks like. That depends team by team, but there should be like, think of for each of these questions, Who's responsible and who's accountable? And I think in the same way, like human labels, maybe the PM is responsible, right? But like, who's accountable for the end quality of the eval? It's going to be the AI engineer. So maybe the AI engineer needs to say, I need 1,000 labels, you know, but the PM has to go figure out how to get those labels. Like, that's the split in responsibility and accountability. The same goes for what happens when the eval is good, but the human disagrees, like, If the AI engineer is signing off on the eval, but knowing that there's going to be, you're going to have to go back and iterate on that, like who's responsible for making that call and then who's accountable. And then the same, same goes for what happens when the eval is good, but the business metric goes down. I think in this last one, I'm pretty confident the PM is going to be responsible if the product suffers. But I think that kind of just reiterates how important it is to get that eval right in the first place. And so like, all of these kind of, I would say, like, kind of show, like, you're going to have a product manager in the loop here at some point for each of these, because it's ultimately going to come down to the business metric. It's— you can't really get away from it.

Yeah, no, this is really good. And we're— so a lot of these things, you know, like figuring out why the eval is good but the human label disagrees, or, uh, figuring out why the eval is good and the business metric goes down, that often requires a fair amount of data analytics, what, you know, some people might call data, quote, data science. Yeah, whatever you want to call it. You know, how does that mean that you need a data scientist? Does that mean that PMs are also doing the data science work? Are the tools there yet where data scientists can do a fair amount of data science work? I've seen a lot of variance in the skills. Where are we at today in terms of expectations and skills of product managers in this dimension, or how do you feel it should be?

It's a good question. I mean, I think we, like, I'm going to be a little biased because I'm kind of working on a tool that helps with this, but I'll try and like pull out some principles for that too, for like, you You don't have to use this tool. I, I, I don't, at the end of the day, whether or not you use the tool I build matters less than like you walk away with the right principles. I think you ultimately, you want a tool or a platform that shows you this single pane of glass between your eval metrics, your human labeled and your business metrics in one place, because you want to be able to plot those correlations and whether that's a data scientist or data analyst that's helping you get the data in the right place and build those correlations. I think that depends if you have the resources for that. And if not, if you're a PM and you're ultimately responsible for the business metric, you might just need to roll up your sleeves, right? Like that, that's part of this as well. It kind of comes back to like the earlier point, like you, you should be the bull in the china shop, like breaking things down to make sure that you have the right data in place to answer these questions. And if that means you have to hire someone or you do it yourself, like, I think that's—

Yeah, let's linger on that for a second. Can you mind scrolling up to that Venn diagram, like the one where they kind of overlap heavily, where it's like more about skills? Yeah, this one.

Yeah.

And so, um, actually I've been talking to Shreya a lot about this, and one thing that she said that stuck with me, that I thought seemed right to me, is like, hey, it's not about job title anymore, it's just about You know, it's about roles. Yeah. Or like you might have to fill many roles. Right. And so this is not necessarily about like, this is not like a war. It's not like some adversarial way people might think, oh, it's like PMs versus AI engineers. Like who's like some turf. But I feel like everyone maybe like, you know, it's like engineers will have to do product management too. Right. It's like it goes in the other direction.

It goes in the other direction as well. Yeah, it goes, it definitely goes both ways. Like I kind of view this more like You remember, okay, I'm dating myself here a little bit, but like, you know, back in the day, if you have like baseball cards, right, you're gonna have like skills on those baseball cards, right? Like, it's not like a pitcher never bats, you know, like you have to be able to do multiple things on the team. And there might be areas where you're stronger and someone else is stronger and you complement each other, but it doesn't mean that you never do the other person's job or vice versa. And I think that that's the level of like, PMs, engineers need to be comfortable with this kind of overlapping territory, you could say, of like who does what.

And it seems like, okay, so with AI coding stuff and AI, you know, whatever, um, Claude Code, Cursor, it seems like a lot more tractable for a product manager to, you know, roll up their sleeves and do some of the stuff that an engineer was doing previously. What about in the other direction though? Like, how can, is there some tools or does AI help the engineer become a better product manager as well?

Absolutely. I mean, I think, um, the, the interesting thing here is like product management is in a way, it's like talking to users, synthesizing data and trying to figure out what to go and build. And the good news is that for a lot of companies that a lot of that data is democratized. So you have tools from the sales team like Gong, which you can go and, you know, if you're building B2B applications, you can go and listen to like, what are your users saying? You can build tools that help you get customer data insights, like, you know, looking at session replay data. So like these tools are, are, you know, they're meant to be used by everyone on the team. And the same way that, you know, we at Arise, we're trying to build a tool to help you take your eval data and correlate it to business metrics. There's tools that help upstream of that, which are take signals from the market and turn it into product insights. And so I think—

So am I understanding correctly, like, okay, so like in this course, you know, we focus a lot on looking at your traces, looking at your evals, quote data. And what I'm hearing from you is like, as a product manager, you're already like, you know, you should be looking at data of other kinds, like the sales data. That's where you're getting— that's like a lot of the the work is like synthesizing all of this like sales information, customer conversations.

Yeah. Can I, if I could double click on that one, it's like, at the end of the day, does it matter if your chatbot is friendly or accomplishes that your goals, if it's not actually doing the conversion that you want it to have, you know? And so like, do you have the right data in place to say our chatbot is completing cases or it's converting users to a purchase? And is that represented? Because that's not going to be an eval, right? Like that, that signal often comes weeks, weeks later, maybe days later, and you, you still need a way to correlate it back to a single session or some feedback. And so PMs need to be thinking about how to get those signals back in, and so do engineers, because engineer— an AI engineer would be responsible for making sure that we have the data in the right place to answer that question. So that, that's like, if an— if a PM comes to you and says like, hey, can we track this back to conversions to sessions? Like, you might need to think about how that how that relates to your evals and how to plot the two together.

In a way, it's kind of seems unifying because you're again saying, look at your data.

Look at your data. Yeah, exactly. It's the same. It's just what's the data in this case, right? And so, so actually that, that kind of brings me to maybe like one of the last points here to kind of tie this all together, which is let's say you have the right evals in place, right? You have evals in development and production. You now have people speaking the same language, right? Like you've, you've kind of solved the messy in the middle of like who labels is, do we trust the eval metric? And you have evals in this. And then you ship, you ship that product. Well, guess what? It's going to work for more users than it did before, but you're still going to have users that complain. That's just how building products goes. You're going to have users that are like, no, I'm not happy with this product experience. That's what forces you to look at your data. And in this case, you have the same input-output eval label, human label, but you probably have some business metric too here. And you're going to be looking at all of these and you're going to have a much stronger signal of like where things went wrong. Let me just go ahead and pull this out a little bit. Right. So you're going to have like the human label was bad. Maybe the business metric is going down. And then that means that you probably need to go back and iterate on your eval. It means that there's probably something missing here where the eval didn't catch something and you want to go back and iterate and improve on it further. And you're never really going to get, I mean, this is kind of a bit of a loop, but the good news is that as you kind of squash more examples where the, the product didn't perform well, you're going to have that eval metric going up, your business metric going up ideally, because these things are now correlated. And when they aren't going up where they disagree, you have a system and process in place to go in and figure out how to iterate. So that's really what I think that for me, a little bit of what the takeaway is, is that, you know, you're really trying to pull people together to look at the same data, make sure it matches what your business cares about and the right people are there kind of asking these questions and answering, you know, kind of who's accountable for what.

Is there any companies that you think are doing this really well? I know it's kind of a hard, difficult question because we're so early and, you know, maybe we don't want to pick favorites. Um, yeah. Anyone come to mind offhand?

I think, um, I can maybe like, you know, rather than like specific companies out there, I could probably list like what I've seen like common patterns of what those companies look like. And they're actually pretty, they're pretty, um, like they do really come to mind because they're outliers a little bit. So there's maybe 2 or 3 traits that come to mind. One is you have PMs that sort of look more like this, but more overlapping, meaning you usually have like technical PMs in the product because they know what data they need to have in place and how to get it. And they're usually engineers that transition into product management, though we're seeing a lot of PMs also starting to trend this way. but it's like you, you have a strong technical AI product manager that knows, okay, here's how to pull this data together and have it correlate. So that's, I think, like a secret, like a superpower. It's like sometimes I talk to these PMs and I'm like shocked that they're product managers because they're not just thinking about, you know, PRDs, they're really thinking about code. And that's a huge lightbulb moment. I think another one is, um, a well-scoped AI product. So oftentimes, like where you see like products in the market kind of, you know, sort of not do so well is they're trying to do too much. And you, you see people trying to throw more agents at the problem. Like, let me just try to like build more complexity into the system. The simpler your AI product is, the higher likelihood that you're going to actually have the right data in place to help answer these questions, to at least get started. And I think that's really where we're at. Like, you don't need some overblown abstraction or a crazy framework. Like, Start with simple principles and be able to track when things go wrong. I think those are usually like two principles I see really stand out for me, um, with strong, with strong AI products right now. AI product teams.

Um, the last question I have is, I think we have heard a lot about how PMs can be more technical. I mean, not just from you, just in general, in the ether of like, hey, this is how everyone can start coding. This is how everybody can like, you know, whatever. What are the, are there any like good resources or some starting point of how like engineers might be, uh, can train themselves to be like, have better product thinking, act more like PMs in this like way so they can kind of, you know, get towards this like overlapped Venn diagram?

Yeah, I think that that's a really good question. And I was talking to, um, we were talking to Brian about this, right? Like in the, in our last like lightning session, he kind of expressed the story of like when he was a tech, like a lead, like a team lead for a product. And he was trying to do it all. And I think, uh, you know, he was responsible for like going zero to one, shipping the end product. And where I'm kind of going with this is like Brian got really, really far doing that until he just had too many customers and he couldn't really scale himself much more. And he was like, I'm, I'm talking to too many customers and I can't write code. I would urge engineers and tech technical leaders to think of themselves in that way, which is like, Even if you have a PM on the team, like offer or help split the PM's responsibilities, but think of yourself as a product person and keep doing that until it actually starts to eat into the time where you can't do both jobs anymore and you have to like figure out how to scale yourself further. Because the more you do that or think that way, like it's just hands-on sort of, uh, experience, to be honest, a little bit of, you know, you have to talk to customers, you have to figure out what's real and not, and you kind of develop a little bit more skill there. Um, I'm biased. There's a ton of like great resources out there, like Lenny's podcast. Um, you know, Akash is another great person to listen to. There's a bunch of these like product thinkers or leaders that kind of help put out content, um, that helps you like use the same vocabulary. But that's really more, I would say that's like more for style points because it's, you still have to do the job at the end of the day, um, to really learn what, what product thinking looks like.

And are you still teaching a course in the future coming up? Can you tell us a little bit more about that?

Yeah, absolutely. So kicking off, also kicking off a Maven session or Maven cohort here. This one is, I would kind of call it like a lot of this information, you know, is you could go to YouTube and like process it or go to like, you know, I think you've done an awesome job with the course reader of evals, and it's an amazing deep dive. This cohort is going to be more of a playbook for how to help you write your H1 or H2 AI strategy roadmap. So think of this as like personal training versus going to the gym and figuring it out on your own. Um, so I'm going to impart and at least show what we've seen those really effective teams do. Like, what are the processes that they follow? What do they get right? Um, it's going to be a fairly, fairly small cohort to start with. Uh, but, uh, yeah, you know, feel free to email me for a, you know, discount code on this one too. Would be happy to, to share that. I, my goal is to build something that your employer is going to reimburse because it's just so obvious, like, you know, that this is valuable for you to learn. Um, and ideally build a community around as well. So yeah, that's the prototype to production. Uh, the AI PM Playbook is the name of the course.

Um, how should people email you? Is that, I don't necessarily, how do you want to distribute that email or how can people get in touch with you for that?

Yeah, so you can just email me at amanaiproduct@gmail.com and I'll throw it in the chat as well. Feel free to email me with any follow-ups or reach out to me on LinkedIn as well. I tend to respond to like anyone that reaches out to me. So happy to be helpful for anyone that's watching this asynchronous as well. Like just reach, reach out to me on LinkedIn. and I'll drop— we can maybe send out the Maven link as well, um, for folks that want to follow async.

Definitely. Uh, yeah, it was really great having you. This is really useful. Um, definitely answered some questions I had. So yeah, thank you for coming on.

Awesome. Thank you for having me. And yeah, let's get more people labeling the right data so that we build better evals and have better AI products out there. That's really what the world I want to live in as well.

Yeah, I love it. All right. Thank you.

Cool. All right.

Thanks.

Thanks.