How to Ace AI Eval Interview Questions (for Engineers & PMs)
In this talk Hamel Husain explains how AI evaluation (eval) tasks have become a common component of AI engineering and product manager interviews. He outlines the typical interview format—a short‑time take‑home where candidates receive a dataset and are asked to build an evaluation pipeline—while warning against common traps such as blindly using LLMs to “find all the errors.” Husain proposes a concrete report structure (executive summary, data analysis, experiments, results, learnings, next steps, and tool disclaimer) and stresses that the bulk of the effort should be spent scrutinizing the data: spotting missing, inconsistent, noisy, or incomplete labels and thinking through how to handle absent guard‑rail examples. He advocates resourcefulness by augmenting limited data with external datasets or synthetic data, and by documenting those choices. The talk then details how to design experiments, present them in tables with baselines, metrics, and trade‑off commentary (cost, latency, over‑fitting), and how to interpret or critique poorly‑designed metrics. Husain also covers how to communicate findings—favoring concise written reports over flashy UI slides—and differentiates expectations for engineers versus product managers, noting that PMs are evaluated more on data‑driven reasoning than on raw metric computation. Finally, he points interviewees to a free, 1,000‑hour‑derived eval guide (bit.ly/evals‑guide) and advises using available tools (coding agents, eval skills) without over‑relying on them, to demonstrate unique thinking.
Chapters
An increasing number of jobs have an evals component to it. Like when I look at jobs online, around like half of the AI jobs I see nowadays mention evals. These are interview tips. So just to give you a little bit of background, I do consulting around evals. A lot of times if you're asked to do something around evals, you get a take-home or you're asked to do something in the office. They give you like 3 or 4 hours and you do an interview. So the way it usually goes is you get a dataset and you're asked to do some kind of evals. I put this meme here. Just please, for the love of God, don't do this. Don't just ask Claude to find all the errors. It's very tempting. This is where a lot of the interviews are designed to booby trap you if you are going to do this. A hypothetical example is TreeBuddy. So let's say there is a TreeBuddy app. The TreeBuddy app allows you to take a picture of a tree and understand if that tree is healthy, how you should care for that tree, etc. The app has like different components. You don't want people to abuse the thing, so you want to make sure the inputs are actually trees. People can't like do things with your application, open you up to liability. And you want to give people like facts about the tree, any health findings, a summary of what they should do, and like along with recommendations, etc. In this particular example, you might get something that looks like this. So you get a dataset. That dataset usually has different components to it. You'll get some labeled data. Let's say you'll get a bunch of trees. You'll get some labeled data maybe around the guardrails they want you to build, examples of bad inputs like where it should pass or fail. Oftentimes they won't give you any labeled inputs at all. There's going to be a lot of traps. In the data. And I'll give you some examples. You'll be asked to potentially do some data extraction. Here we have photos and we're going to do data extraction of some kind. And data extraction is a favorite thing that people like to do in evals. It feels more verifiable because you can compare data. They can give you some data that they want you to extract and then compare the data you extracted to some golden set. And then there's unlabeled data. In this case, you might have the summary and recommendation just like text. They may be labeling that, they may not be labeling it, but usually there's some form of unlabeled data somewhere. And then a lot of times what you are asked to do as a PM or AI engineer is like build an agent that can perform well. And they'll give you some evals, they'll give you some metrics that they want you to like think about. Okay, so this is what I recommend. You want to produce a structure. This is the way I like to structure it. I like to have like an executive summary up front. I'll go through these sections. Before you talk about anything like how you built the app, what the results, whatever, talk about the data. You always want to focus on the data. What data exists? What issues are in the data? Then you want to talk about experiments you ran. You want to summarize the results. You then want to talk about what you learned and then next steps. And you always want to then have a little bit disclaimer of any AI tools you use. Depends. It's always good to have that. So the executive summary, I'm not going to spend too much time on this. You want to keep it to a third of a page or less. You want to just say like what you built and the results. So think about like a short paragraph. And there's going to be some repetition in this like artifact that you create. So you want to go directly into the data. And this is like where the meat of it is. A lot of people skip this section. This is like the most important section. This is where it separates people who understand evals and people that don't. I say spends 50 to 70% of your time here. And here are some things you want to look at. You want to look for things like missing or inconsistent labels. You want to look at insufficient label data, like maybe it's not labeled correctly or maybe the labels don't make sense. Never assume a dataset you are given in real life and especially in an interview is, quote, correct, because the person is trying to simulate your ability to be skeptical and like really look at the data. And that's what they're doing here. And also you want to think about how noisy is the data. And these are all like very related, but you basically want to see what are the issues in this data. And let me show you some examples. So let's say here's an example of a dataset. There's 120 examples, and there are some attributes of this data, like some metadata attached to these photos. And they have like species, like what species the tree is. Every example is labeled, but only 96 of those examples have a health finding at all. Maybe only 24 of them have height. Maybe only 8 of them have age. I just made this up. And then like we talked about guardrails, Let's assume they just give you a dataset, but they had a guardrail pass, or is this violating a policy of like what pictures can be input? And they all say false. None of them violate. So the whole dataset doesn't have any violations at all. You'll get examples like this. You might be wondering, well, how the hell am I supposed to build this if I don't have examples of what is pass and fail? That is exactly the problem you will have to solve. That is the mess that they're going to throw into here. If you just try to benchmark this, you'll get like 100% accuracy on input policy because there's not any bad examples. So they're trying to test you, see like how you handle this data where you get like incomplete data. So one thing you might want to do even before doing synthetic data is— and synthetic data might work— what I like to do is be as resourceful as you can. So what I mean by that is if you're given a problem with this tree thing, maybe there is like a trustworthy dataset that someone already has for some of these dimensions. If you think about it, like it's kind of hard, right? Like height, age, like where are you going to find that? You're just going to synthetically generate pictures of trees and know what the height and age are? It's a little bit tricky. So you want to try to show off your resourcefulness. And what I would do is think about outside data where possible. Not always, but like in the outside data doesn't necessarily help you solve everything, but it can expand the dataset and sort of say, can I have like a reference dataset that I can test with and use the dataset that they gave me as the test set as like the final like readout on how it's going. You're gonna get this incomplete data. You might think, oh, that's nice, Hamel, it's trees, whatever. The reason I'm like constructing this hypothetical example with the trees, I found that like 80% of take-home exams exams and examples, you could always leverage outside data because they're trying to give you an example that's like amenable to an interview a lot of times. Not always the case, but always think about how you can be resourceful. Is there any other data that you can use to augment what you're doing? And it's like, okay, these are 2 datasets actually. There's this tree Curio tree data on Hugging Face. There's also this research data archive on trees. And for the guardrail, we mentioned synthetic data. That's exactly right. I would actually create these negative examples here. And these are not real people and real things. You can see that this is AI generated, but it's better than nothing. Again, you're trying to show off your resourcefulness because like you are given limited resources when it comes to data and you are trying to be creative on how you fill those gaps. Sometimes with synthetic data, maybe sometimes with external data, whatever it may be. You try to choose like what is the best proxy here. But most importantly, you want to actually look at the data and you want to find things like inconsistent labels. How would an inconsistent label look like? The tree on the left, it says species is English oak. And then the tree on the right says species pine, period, long needles, orange bark. Then the height you see is not normalized. So you have like height 12 meters. I don't even know if that's a reasonable height for the tree, honestly. And then there's height 50. 50 what? God knows. I don't know. If it's a good evals interview, they're going to give you this to try to frustrate you. You have to think about what to do here. You have to think about either label this yourself or get an LLM to try to do your best job to normalize these labels. The whole idea of the artifact that you give them is like you discuss it. You say like, hey, this is the issue that I found in the data and this is what I tried to do to overcome it. You might not be able to perfectly overcome this. Now we do have that dataset where maybe we can have more trust because it is normalized, those external datasets on Hugging Face and the other one where everything is normalized. And you can say, hey, like because this data had these problems, I use this dataset as a proxy to like try to get things started. So again, like it's a job interview and you want to show that you're paying attention and you're looking into the details And you're understanding the data. And you're not just haphazardly pointing your agent at this. They're trying to simulate like what actual product work is going to look like, what you are going to encounter. And then like you want to also pay attention to noise. You can tell I generated this AI, but I'm trying to drive home the point. You want to see these examples and you'll see there's real trees and then there's bonsai trees. These like miniature trees. Do we want these bonsai trees in the dataset? A lot of times there'll be things in the dataset that shouldn't be there. You want to call that out when you write up this artifact to say like, hey, I took these out of the dataset because it didn't belong. Or you have to make some product judgment like how you want to handle this. But there's these traps that you want to think about. So when you look at this data and you see what's going on here, you will find issues. You just want to summarize those issues. You want to like maybe have bulleted lists. What do they look like? What's going on in this data? And you want to use your skills of communication to talk about what the data issues are. Okay, so next, moving on to an experiment. So what are experiments? Experiments are your attempts to make this agent better at solving this problem that they posed to you. You want to frame it like this: what is the goal of experiment, what you tried, and then any methods and findings that came out of that experiment. An example goal could be species identification of the trees, and you want to classify the species correctly. What I recommend you do is to have a table of results. They will give you a metric that they want you to express, and it may not be TPR, TNR, whatever. It may be something that they have. Whether you argue with that metric is up to you. We'll get to in a moment. Sometimes you might want to argue with that metric, but only if you think it's wrong. And I'll show you an example of where the metric might be a bad idea. But what you want to do is show a table of different experiments that you ran. And you can organize the table in any way you want. It doesn't have to look like this. But you want to say, hey, I tried this thing. So for example, in this table, I tried GPT-6 Luna with just a prompt. I tried GPT-6 Luna with this external tree database where, like, when I look up the answer, I also try to get the most similar tree in the database. And I use that to inform my answer. I'm just making this up. V3 is like, hey, I'm doing the same thing as V1. I just changed the model to SOL. And V7 here is like using automated prompt tuning like JEPA or something like that. And the idea is you want to show the different things you tried. Like how did you try to improve this? You want to have a baseline. So you want to make sure you have like a simple baseline. That's why I have on this slide prompt only. You want the simplest, dumbest thing you can as your baseline. And then you want to try different ideas you might have to improve your agent. And there's no formulaic way to go about this. This has to be informed by your judgment, what things you want to try. This is all an opportunity to showcase your knowledge, and it's a nuanced thing. If I had to do one of these today, if Jev just came out, I'm like, let me try Jev. It depends. If I'm interviewing at Anthropic, then I may not use Jev because I'll just use their stuff. You have to think about like what they want to see. They might give you parameters of things to try, or they might tell you use a certain SDK or use this, use that, whatever. Think about having a table of results where you can quickly show things you tried, and then you want some narrative under the table. I tried this, this is what this means, I found this approach to be good. You want to show some thinking of trade-offs. When you have a table like this, you might have something that has like the highest score, but maybe that's not the one you want because it's slower, it's more expensive, it's more complicated. And what kind of the next level is to like talk about, okay, like I actually like this certain example because I think it has the best trade-off in terms of complexity and performance. And there's also, if you notice on this table, there's this validation and test. People won't tell you to do validation tests. You want to pay attention to overfitting. See this like V6 row here has a very high test score on the validation set, but the test set score drops significantly. So something is being overfitted. Like the score is not consistent and we don't like that. So these are the kinds of things you want to discuss. It's not just about score maxing. It's like bring your judgment and talk about, oh, like this is the kind of tradeoffs I think about and this is what I would like. You want to have your methods underneath the table and you want your findings. So like an example finding would be SOL and RAG seems like the best trade-off. Astra is overfitting and automated prompt tuning did not help. And again, you're trying to show off your judgment, your ability to think about trade-offs, and also showcase that you know a little bit about different approaches to improving the agent. But you want to write it. Don't let AI do all this stuff for you. You're going to get in trouble. You're going to be like this person in the photo. Don't do that. Now sometimes this is a kind of insidious thing to do, kind of borderline evil, but it will happen. They will ask you to do things that you shouldn't be doing. For example, you have this tree. Never mind that the tree— the picture sucks. Can't even see the whole tree. That's a problem. But okay, like your app produces this summary. The brown leaf edges mean your tree is drought stressed. Give it extra water this week and watch for more browning. It's clear you've been neglecting your tree. Let's say the interviewer gives you this direction: grade helpfulness and tone on a scale of 1 to 5 and combine the helpfulness and tone score into a composite score from 0 to 100. You want the score to be actionable and interpretable. Like, what the hell is this? This is a little bit trickier. It will come down to this. Now, sometimes this is a very insidious question meant to trap you. Sometimes they don't know evals. They're interviewing for evals and they don't know evals. I would always just compute the score, but then also explain, hey, this is not the way I would do it. I would think about refactoring this metric to blah, blah, blah. That always goes over really well. There's 2 possible outcomes from that. One is You avoided the trap that they set for you. You're going to teach them something. They're going to be like, oh, you know, that's a very thoughtful comment. If you explain properly, like why you have to be nice about it and you have to explain yourself, give the reasoning behind that, they will learn something and that will put you at a completely different level than anyone else doing this interview. And that's what you need. You need to be at a completely different level. AI interviews nowadays or job interviews in general, they just have way too many people they're considering and way too many applicants. So like from this specific example, what are some examples of evals that I would create? I would say like, okay, is a full-length picture taken? Is the watering advice— maybe it needs to check soil moisture. Like, why are they giving that to me? Let me make sure it's grounded. No blame or scolding, saying you have been neglecting your tree. Why say that to you, neglecting your tree? That's, that's harsh. I would describe why these are better, because you don't want to be caught in a situation where your interviewer is like, how do you know that the score is good? So experiments, You want to put it all together. You want to have, like I said, the goal. You want your table of experiments, and then you want those methods and findings. Expect to have lots of experiments. You would want to break those experiments down by different features that they want you to tackle. They have like guardrails. They want you to classify the species. You get like health recommendations of the tree. Okay, there's a lot of things going on here, and you want different experiments for those different areas. And then it's going to results. So this one is quick. You want to just synthesize your experiments. So you're going to have all these experiments that you run. You have many different tables. Probably one experiment per feature. So you're going to have a lot of information in that experiment section. What you're going to want to do is like condense it to results. You want to say like, okay, what is the winning combination of these experiments? How would you go about it? And you want to synthesize it and say, okay, these are your recommendations on how you proceed sort of based on all these experiments. You want to reiterate like what the baseline is, what the final eval scores is across these different evals they gave you. And you want to also think about the cost and latency trade-offs. So remember I talked about trade-offs. It's not just benchmarking or getting the highest score you can. There's opportunity for you to demonstrate your product thinking and your judgment. I would go as far to say you should have something in your table of results with a higher score that you decide not to do. Does it have to be like so unrealistic, so heavy-handed? It's like expensive, it's overcomplicated. It's really good to have that so you can showcase this quality of yours of judgment. It's like, no, I'm not going to go for the highest score. That would be bad for my product. This other alternative is better because it's just a little bit lower, but it's so much faster, so much cheaper, so much more stable. And then after results, I would talk about what you learned. And you could— there's some repetition in this, and some repetition is good. People don't read. So there's some amount of repetition that you need when you submit these artifacts. And so what you learned kind of should talk a lot about like what worked well, like what experiments you tried, what didn't work. So like one example of what worked well, maybe synthetic data was really effective. Maybe this external data was effective with retrieval. Maybe, maybe you caught overfitting. Talk about all the things that you learned while you were building this application. Talk about what didn't work. Are those experiments like kind of synthesized? Like, oh, okay. Yeah, maybe prompt tuning didn't work. Maybe larger models didn't work. Then talk about limitations. So what I do here is like repeat any bad things that you were forced to do. Maybe they were telling you to like take like 10 generic metrics. And combine them into a single score. I have seen this on a real take-home exam by very reputable companies of like, hey, let's take these 5, 10 metrics that are super generic and combine them into a single score. In cases where you are asked to compute something from text data and you don't have labels and you can't get labels, do your best. You might even go as far as say, hey, these LM judge would not validate against human labels because of like this, this, or other limitations. Try to label it yourself if you can. But hey, you might not be able to if they give you something, if they give you a dataset. Keep in that when you encounter interviews, it will be a mixture of people trying to set a deliberate trap and people who don't understand evals. They're just getting an AI to generate an eval interview. They didn't think through it. That happens a lot. You still want to showcase your knowledge here. You want to teach the interviewer something. In that case, that's the best outcome. And again, you want to repeat any limitations of the data that you saw or the approach. Okay, so this is like where you exercise some humility and say, okay, like, this is kind of like where we were limited. And then always like to have a next step section. You want to put Product head-on is like, oh, okay. Like if you had more time based upon what you learned, like what are ideas you would have for improving performance? And it is always the case, 100% of time that you can have better data. Everyone can have better data. And so you should always mention, oh, like I would try to get better data on XYZ ABC. And also I like to talk about product improvements. This is again another opportunity for you to showcase how good you are by thinking outside the box. Box and helping that person envision working with you. You talk about, oh, you know what, you gave me this tree app, I would actually redesign the app to do this thing instead, or like, you know, have these other functionalities that will make it easier for you to evaluate. But okay, to make sure if you do this, if you talk about better data or product improvements, you discuss the details. So don't just say better data, talk about like what exactly do you mean, like what's wrong with the data, what kind of data would you get exactly. And then like product improvements, like what, what are some product improvements you might try to make? And that's it. That's kind of my synthesis of how you should think about framing the problem if you're going to do an interview like this, if you're going to think about evals.
A few questions there have been around the experiments that you have been talking. How does it differentiate with A/B testing that normally software engineers do?
I would say don't worry about A/B testing unless it's a data science interview. They're not going to usually do that. They're going to give you a dataset and ask you to compute some evals. And compute some metrics.
Will it be helpful to present the results in, in kind of like UI?
Be very careful with that. So it depends what they ask for. Usually they ask for something like, hey, please give us a report in a PDF, please give us a GitHub repo. Do you want to follow that? Be careful about UIs presenting stuff. Like, you don't want to become a slop cannon. Easiest way to turn someone off in an interview is becoming a slop cannon. The reason I say this is like, I can't tell you how many slides and stuff I see all day that I can't read. And so just don't do that to yourself. Write it thoughtfully.
Whatever you have covered so far, it was more of like very much technical topics for you. So what is the normal bar for PMs into the eval space, like interview for PMs?
So I would say like, it's not as technical as it might sound. So if it's a PM interview, they're not going to like hammer you on calculate F1 score. They probably won't even ask you for an F1 score, but they will tell you to like, look at the data and tell us how you might go about this. They might ask you to vibe code something. They usually do nowadays. They might give you a dataset along with that. And what you should do is try to like look at that data, applying some of these principles and showcase like all of the nuanced thinking that you have.
Last question that I can see is many questions I've seen around this mock interviews. Like if there's any platform for mock interviews or just a question to you from my side, like given Claude Code Skills, Claude Skills, Claude recently the eval skills came up and couple of like Codex and other skills. Is this possible to mimic some kind of like, okay, what kind of, given you have some set of data or traces, to mimic what kind of eval process should go around or like in the, in general, the interview process?
Yeah, I mean, you should use whatever tools you can. It's part of resourcefulness. Use your coding agent, use the eval skills, use anything you learn in this class, et cetera. I would like be very careful not turning your brain off and saying like, oh, how can I solve this interview with an agent? Because that is going to cut against you because that's what everyone else is doing and you do not want to look the same as everybody else. Like the person needs to read your report or listen to your presentation and hear something different. And the way you do that is this like way that I showed. Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals. You don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. Go to bit.ly/evals-guide and we'll send it to you.
