From Noob to Automated Evals In A Week (as a PM) w/Teresa Torres

Welcome everybody. We're really excited about this talk with Teresa Torres, who was a student in our first cohort. Teresa Torres is a legendary person in the product management space. Um, she has trained, I think like over 70,000 people or something like that on how to do product management effectively. She's the author of this book, Continuous Discovery Habits, which is right behind her. You'll see it. You should definitely check that book out. It's very, it's like one of the canons in product management, uh, from my understanding. I'm not a product manager myself, but every product manager tells me that. The reason this talk is so exciting, the reason why you should pay attention to this is, hey, like Teresa, she's not coding every day or anything like that and wasn't like necessarily doing evals, but she had her own approach. Combined with her product management expertise, and she has a unique take on how she went about things and has found a workflow that she likes, and she's going to share that today. So I'm going to hand it off to Teresa to get her started.

All right, I'm really excited to do this. I really enjoyed this class quite a bit, and you're going to see I learned a ton and was able to put a lot of it into practice right away. And I cannot emphasize this enough, I forget when the first cohort started, I think it was in June, maybe May, I barely knew what an eval was, like, and we'll talk about that in the, in my talk. I am gonna share some slides, but I'm happy to keep this very conversational. So please feel free to jump in, Hamel Husain or Shreya with questions, and we'll just dig in. I'm gonna start by giving a little bit of context about the product that I'm building. And sure, Hamel mentioned some of this in my intro, so I'll touch on this briefly. Like my day job, I work as a product discovery coach. It means I help product teams make better decisions about what to build. You might be— some of you I know are familiar with my book. We have a wide variety of courses. In fact, I saw Letty is here. Letty is currently in our continuous interviewing course and has gotten to experience the continuous— the interview coach firsthand. But basically, because I primarily am an educator, I, when I started to think about generative AI and how it might impact what I do, I was thinking about it from a teaching standpoint. And that's the part that's relevant here. So I found myself in March, I actually had to have ankle surgery and I was like on the couch for months and I was like, how in the world am I gonna use my time? And I started this with this question of what does generative AI make just now possible? And I was asking this in the context of our courses, and there was one course in particular I thought this would help a lot with, which is our continuous interviewing course. So in this course, students learn how— this is customer interviews, so not job interviews. And in this course, students learn how to ask the right questions. They practice active listening techniques. We teach a very specific type of customer interview where they're asking the customer to share a specific story about their past behavior. And there's reasons for this. This is a way to collect reliable feedback. I'm a little bit of like a research methods wonk, so I'm like trying to look at what's the easiest way to teach product teams who are not researchers effective ways to get customer feedback. And in all of our classes, they're very practice-oriented. So like in this class, students come to class and they interview each other and we give them a rubric to help give each other feedback, right? So I'm a big fan of deliberate practice. We practice a skill, we get feedback, we then know exactly what to practice. There's one challenge though, like when students give each other feedback, they're not really experts, so they don't give expert feedback. And so the problem I was looking to solve was like, can we help students give each other better feedback on their practice interviews and that led to this question of can we model what good feedback looks like? And so this is the tool that I built. I built an interview coach where students submit their interview transcripts and they get very detailed feedback on what their, how they did on conducting an interview. And the coach actually, this is actually what you get from the coach. And the coach, this is actually a partial response, but we, the coach is basically evaluating an interview transcript on 4 different dimensions. And this is, from the rubric that we teach in the class. So that we're evaluating, did they ask a story-based question? Did they ask the participant to set the scene? That's the 2 that you're seeing here. There's also a dimension about building the timeline. How well did they get the participant to tell their story? And then did they do a good job of redirecting generalizations? And you can, I'm gonna dive into this a little bit 'cause a lot of my evals are about this coach feedback. So this'll give you some context. So you can see here for each dimension, they're getting a score. The score can be keep practicing, getting it, or great. This coach is pulling out real excerpts from the interview transcript and then giving them a tip. So this might be, this was really good. It might be, here's some room for improvement. And it's doing this across all 4 dimensions. So there's 2 sections of this email we're not seeing, but it's basically excerpt, tip, excerpt, tip. Okay, so that's the interview coach. So let's get into—

as I said, dimensions, is it that those— these is like different stages, opening with the story-based question and setting the scene, are those the dimensions?

Yeah, it's— so what we're looking for in the transcript is, did they ask the participant to share a story? We teach that they should ask them to set the scene because this helps with memory recall. So it's a cognitive tip to help them, to help get a more reliable story. And then the building the timeline dimension is, did they help the participant go step by step through their story so they're learning what actually happened? And then the last dimension is, if I ask you, tell me about the last time you watched Netflix, you might start to generalize about what you usually do on Netflix. And so my job as the interviewer is to redirect you back to the specifics of your story. And so that's the fourth dimension that we're evaluating on.

Interesting.

Yeah.

I need to read the book too. This is also, it's already very interesting to me.

Awesome. Okay, so as I was building the Interview Coach, I literally was like vibe checking the coach's responses. Some of them looked good, some of them had some problems, and I started to get concerned, like, how do I know if my Interview Coach is any good? I would make like—

So what did you have in the Interview Coach? Was it like transcript of your book? What information like was your initial version?

Yes. So because we have a course on this topic, I already had a rubric for how to evaluate an interview. And so I started with the rubric that we use in the class, and then I included a lot of the instructional content that we provide to students in the class to the LLM. and like my very first prototype was in a Claude project. I just uploaded a bunch of files to Claude and I was just like, here's an interview transcript, give me feedback. It obviously has iterated and it evolved quite a bit from there. But basically the coach, the AI tool, the coach gets a full interview transcript and then it has a lot of context about what's a good interview, what these four dimensions are. It started as one prompt. It's now like, depending on the content and the context and what errors uncover, it's anywhere from 5 to 9 LLM calls. Um, so it's a pretty complex workflow, and I can imagine one day it will be an agent. Like, what I would love to do is to pull in a student's learning history and then adjust the feedback based on where that student is on their learning journey and get— and move the coach to more of an agent model where it's choosing what dimensions to evaluate on based on the student's history. But I'm not there yet. Okay, so like in my earliest days building this— so thank all— by the way, all this is fresh. I started this on March 23rd, and I really had never built an AI product. I was using all no-code tools at this point, and I just was making I had one prompt, like it started as one prompt and I was, I didn't even know what temperature was when I started. And so as I was making tweaks, I would see some things get better and some things get worse. And I was like, this seems insane. Like product discovery is all about feedback loops. That's literally what I teach. And so I was like, oh my God, what is my feedback loop? Like I need a more systematic way to evaluate this. And you're all in an evals class, so you already know the answer is evals. Um, but I got stuck, and you might know, you might understand why. Like, I started at OpenAI, and they talk about evals, and it sounds like it's just a bunch of input data. And I was like, yeah, but I don't have a bunch of transcripts. And maybe I could synthetically generate transcripts, but let me tell you how hard that is. Like, it sucks. And so I kept looking, and I found like this Python package, and it What was interesting to me here is it started to talk about evals as code and I was like, oh, interesting. That's like interesting. And then I came across Aman Khan's blog post on Lenny's newsletter and I was like, oh, there's different types of evals. And now I got sucked in and that's, we're gonna skip this, you guys know what evals are. That's when I found this class. And I will tell you in the very first session when this model was introduced, like light bulbs went off in my head because this is the scientific method. I teach the scientific method and I was like, okay, I'm in, let's do it. Um, and so over the course of the 4 weeks of class, I literally, instead of doing the homework, I just applied each lesson to the Interview Coach. And this is how I work. I'm a very goal-directed learner. So I was like, I have a project, I'm gonna take everything I'm learning and just apply it to my project. And so again, some of these slides were for a non-experienced audience, so I'm gonna skip past them. So I started by collecting my traces in Airtable. So why did I use Airtable? This is a tool that I'm familiar with. I was already had plenty of new to learn, and this is gonna be a theme of this talk. Like I try to limit how much I have to learn in each step so that I don't get overwhelmed. And so I was like, okay, I see that there's these eval tools, but they look really overwhelming to me. I'm just going to store everything in Airtable. And so that's what I did.

It's an excellent approach. I highly recommend that people do this. It's like, hey, don't try to swallow these tools. Everyone wants to go for tools, but yeah, stay where you're comfortable with in the beginning for sure.

So I think I had— God, where did my original data came from? I think I had about 15 interviews that I conducted. I think I had about 10 other interviews that some of my instructors had conducted, and then I think I had about 10 interviews that some of our alumni let me play with. I was developing this tool, so it was small datasets, and I actually used Airtable interface as my very first annotation tool. So if you're not familiar with Airtable, like, it's a database basically, and then it gives you these like UIs to construct viewers. And you can see here, like, I have a trace down here at the bottom, I have an annotation box, I have my failure modes, and then I can mark it as reviewed so that it doesn't show up again in my annotation tool. Dead simple, right? And if I go back, you can see, like, I've got a bunch of annotated traces, I've got failure modes. This is my— these screenshots are from my very first round of doing any of this. This is literally during the class. And I'll tell you, when I saw this, look at all these failure modes. I was like, holy crap, my coach sucks. And what's funny is that I thought it was pretty good. Like when I was just vibe checking responses, I was like, oh, it's pretty good. Yeah, I was embarrassed at this point.

This is amazing. I love this so much.

Okay, so I then sat down. I remember this vividly cuz it was like a Sunday evening. I'm sitting on my deck. Right next to my husband who's also on his laptop. We're both nerds working away, and I was trying to design my first set of edals. So I was literally like, whenever I code— I didn't talk about my coding backgrounds. As an undergrad, I took some computer science classes. I did a cognitive science program that— where computer science was a big part of it. I— my first couple jobs, I was also a front-end developer, so I did JavaScript script development before there were any frameworks. I did not code for 15 years, and then maybe longer, and then about 3 years ago I started coding again to automate a lot of my business logic. So I run cohort-based courses. Running cohort-based courses requires a lot of like data management, and so I started coding to acknowledge, to do work with that stuff. I'm not really an engineer. I had never worked in an IDE. Like I didn't have a dev environment and a prod environment. Like I didn't know how to do these real things. And so I actually chose to work in AWS's serverless environment because like I didn't wanna manage an EC2 instance. Like I just wanted to like execute code and move data from this app to this app. And so I started writing Node.js Lambda functions and then orchestrated them with Step Functions. So that was my experience when I started. Started this product. And you'll see this influences a lot of my tech decisions. Okay, so it's Sunday evening, I'm like, okay, this is in the middle of the first cohort, I've identified my top failure modes, and I'm like, I need to write my first evals. And whenever I write code, I always start in pseudocode, and I just start like in a text document. And I literally was in an Apple Note, and I was like writing out JSON. And I don't know if you guys have ever used Apple Note, but it turns quotes into smart quotes, which breaks your code. And I was like banging my head against the wall and my husband goes, why don't you use a Jupyter Notebook? And I was like, what? What's a Jupyter Notebook? So I literally asked ChatGPT, I'm a total beginner, what's a Jupyter Notebook? For those of you that aren't familiar with notebooks, this is a Jupyter, this is a screenshot of a Jupyter Notebook. This top part up here is Markdown and then my code is here. Here's what this allowed me to do. I started in a Markdown cell and just brainstormed like different ways to do an eval in kind of pseudo code. And then I could try to write that in my cell block and actually execute it and then see the results. And then what I realized, which was really great about this, is like for one eval, I might have 3 different ideas about how to implement it best. I could do all 3 in a different cell block, run them, and then compare their output. And so that was awesome. I was like, okay, I'm sold on notebooks. I'm gonna show you two of my first evals that I ever created. So this one, I, so we need a little bit of context. Remember, the coach is taking in a transcript and it's sending you detailed feedback on that transcript. And sometimes it'll suggest to you, it'll be like, Hamel, you asked a great question here, but you could have asked this follow-on question. Right? So sometimes the coach will suggest a question to you. The problem is sometimes the coach's suggestions are wrong. So there's two failure modes that came up where sometimes the coach would tell you, you should ask a leading question. Like the question is suggested to you was a leading question. And if you're not familiar with this concept, a leading question is just a question that implies the answer. So it's like you're leading the witness. So that was one of my failure modes. And then my other failure mode that was really concerning to me is sometimes the coach would suggest a general question. So it would say something like, you should have asked, tell me about your typical morning routine. Well, we don't teach that because that leads to really unreliable responses. We teach, tell me about your morning routine this morning. So the specific, not the general.

And so for someone that's new to notebooks, and stuff. Like, it seems like you organize your notebooks quite well. You have like headings, you have a little bit of documentation, and you kind of print it out to show yourself what it does. This is like, yeah, any good practices?

So here's what I'll tell you. I'm not an engineer day to day, but like, I did— I took a lot of computer science classes at Stanford where they hammer like elegant code and well-documented code and like I feel like I have a pretty good foundation in like good engineering practices. And then as a product manager, I always worked at teeny tiny startups where like I was involved in all the engineering decisions. So I think I'm a very technical product manager and like actually now like 4 months in, I'm ready to call myself an engineer and you'll see why as we get further along.

Amazing.

Um, I'm, it's actually been really fun like embracing my engineering skillset. Okay, so these two evals, these two error cases, the coach suggesting a leading question and the coach suggesting a general question, as a teacher were catastrophic for me. These are two of the things that students really struggle with, and I did not want the coach reinforcing it. But if you look at this code, this one on the left looks really complicated. It is not complicated at all. What's happening here is this is a prompt where I'm explaining to an LLM what a leading question is. And then it gets sent a list of questions and it's being told if any of these questions are leading, fail the test. And if any of the question, and none of the questions are leading, pass the test. All this code down here is just Python to call a production Lambda function, so you can ignore all that. So this is an LLM as judge and I iterated on this a lot. I tried like sending all the questions at once. I tried sending one question at a time. I tried sending the full coach response versus just the questions. So that's what I mean by I tried different strategies for how do I get this eval to be fully aligned with my human labels. And then this eval, the general question, I actually banged my head about how do I teach a LLM what a general question is? Do I have to like give it the whole context of a specific story and the human tendency to generalize? And actually I realized that there's just a small set of keywords that tend to indicate a general question. And this turned into a code assertion. I'm just looking for red flag words in the questions the coach suggested. I've extended this list of keywords over time, but what's funny about this very elementary eval is it has a really high alignment rate. Like it never has false positives and it catches almost everything I catch.

I'm so happy about this. You know, you nailed it because what you want to do with evals is you want to have the right cost-benefit. And the fact that you had this intuition like, hey, this can be a codebase eval, a lot of people would miss this, honestly. They would say, oh, I don't know, let's have an LLM do it. You had the good insight of making a codebase eval.

Okay, Hamel Husain, I'm glad you highlighted this because this was the moment where I was like, holy crap, who's going to do this in real life? Like product managers, designers, engineers, like what's the right role? Because the reason why I could identify this insight is because I've been teaching this stuff for years and I see the mistakes that students make. Like this required domain expertise. And so like if I just left this to an engineer who wasn't me, I don't think this would've happened. And so this is still an open question for me as a discovery coach that teaches cross-functional collaboration. To product teams, I'm like, oh, we gotta get good at working really closely together because not every product manager is gonna write code like I did, but I think they need to be involved in the design of their evals.

Definitely. I can almost guarantee you with almost 100% certainty that an engineer would not have, would've tried to make an LM judge out of that and not have. Yeah, totally.

Yeah. So these two are literally the first two evals I wrote. I love them because they're, still in, they're still in production today. Like they're still really valuable. I've actually been able to reduce these errors quite a bit. And so like I was, this sold me and I did this on that first Sunday night. Now I wrote these on that first Sunday night. I didn't have any infrastructure for like how do I run them against all my traces and how do I score my judges? So we'll get to that. Okay, so the other story I didn't tell you is on Sunday night when I started all of this, I had never even seen Python, but I think my husband works with Python and he might have told me that Python has all these visualization tools. I think he did. And also in class and in the Discord, everybody was talking about all the cool Python visualization tools. So my notebook is actually a Python notebook, which still blows my mind. And how did I do this not knowing Python? I literally had ChatGPT open and was like, here's what I'm trying to do, can you help me figure out how to do this? It would present some code. I won't run code that I don't understand. So I literally would go line by line and be like, what does this mean? What is this doing? And then once I understood it, I would copy it into my cell block and run it. Okay, so ChatGPT plus my Python notebook helped me import like my traces. So I exported a CSV file from Airtable, I imported that CSV file and used like Pandas to look at it, and I created a cell block in my notebook that ran all of my evals against— at this point I had 1, 2, 3, 4, 5 evals against my traces. and oh, the other thing I did in my notebook, this I think might be helpful for people. These two evals, they don't actually take in the transcript. I actually do some transcript processing to help make my evals more efficient. So one thing I noticed is all a lot of my failure modes were related to the questions the coach was suggesting. So I had a helper function that it's just code that like takes the JSON response from the coach and pulls out all the coaching tips. That's just code. And then I have another helper function that I think is an LLM that pulls out all the questions from the tips. So then my evals don't have, especially my LLM is Judge evals, they don't have huge input tokens. It's, I've just constrained the context to what that eval needs to assess what it's assessing. And actually, this has saved me a ton of money and makes my evals really fast because when I'm testing and iterating, I can run all my helper functions once for all my transcripts. And then as I change an eval, the input data isn't changing, so I don't have to rerun all those helpers. And those are like the expensive LLM calls, whereas my evals end up being a lot more lightweight and a lot cheaper.

This is amazing. This is so amazing. Because not only are you saving money, but like, you're also making the eval better if you're constraining the context to like, only what it needs to know. If you've done that, that's pretty advanced. And that's really great that you like, did that work. And people would call that quote, context engineering, the new hot term. You have been doing context engineering from the very beginning, which is excellent.

Yeah. And what's funny is that this tapped into— I did this originally just as like decomposition. I was like, oh, this eval and this eval need the same stuff. I'm gonna separate that into its own function. And it was just like a separate cell block in my notebook. And then I realized that, oh, this enables different experiments I can run. Like what if I give this eval the full coaching output? What if I give it just the questions? What if I give it one question at a time? And that's what really allowed me to tweak and figure out which strategy was most aligned with my human labels. And I literally did all of this in a single notebook. Okay, so then I, like, in my notebook, I worked with ChatGPT and was like, okay, I have evals, I have the CSV file, how do I run all my evals against all my traces? It helped me create a, a code block that does that. And that code block outputted what you're seeing here. And this is basically my eval results, and this was my very first run. So I have 1, 2, 3, 4, 5 evals, and the traces are the rows. And you can see at a glance, I can start to see how I'm doing. So those first 2 error cases that I was most concerned about, they're happening pretty frequently, like I think about 20 to 25% of the time. And then you can see in this middle one, it looks like, wow, this fails all the time. And this one on the right seems to be a huge problem. But as I dug into this, I started to realize some really interesting things. First of all, this one on the right, when I created this eval, what this eval is, is I have 4 dimensions, right? Sometimes a dimension is greedy. It expects every question in the interview to be about its dimension. So my setting the scene feedback would be like, this is a terrible setting the scene question, but it was a really good building the timeline question. and the interviewer didn't do anything wrong. And so this error mode is it expects too much setting the scene. So like they asked good setting the scene questions and then they moved on to building the timeline, and the coach was critical that they asked building the timeline questions. And so it looks like this is happening a lot, but what I did to dig into this is I, in my notebook, I have tools that allow me to look into each of these failures and to give and examine the reason the LLM gave for why this failed. This one actually is a code eval, so that's not quite right, but it could, it, I could look into the output of the code and then I could look at the individual trace and my human labels. And again, I'm doing this all in my notebook. I don't have screenshots of that, but I'm doing this all on my, so I like basically as I dug into my data, as I had questions, I would go to ChatGPT and be like, Okay, it looks like my evals gave this and I gave this. Can you create a cell block where I can compare my human label to the eval label? And so then I just iterated with ChatGPT to turn my notebook into a data analysis tool. And what I found here in this last column, it actually was a problem with my human labels. Like my eval was better than my human judgment. And it's because to write a specific eval for this error mode, I had to get a lot clearer about what I was evaluating. And I, like, codified that in code, but I hadn't codified it in my human label, like, rubric. And so this was a case where the code was better than I was. And so, like, I had to change the way I did human labels, which I thought was really interesting.

This is so amazing on so many levels. The one thing that really impresses me is you actually took on the mindset of a data scientist. Pretty good, like, putting on the hat of, okay, let me investigate the data, let me look at it in different ways, let me figure out what's wrong, and like really dig around with this sort of investigative approach.

Yeah, and this is, I think, where notebooks shine. Like I wrote little pieces of code to like visualize things a little differently and to look at different data. And in some ways it's throwaway code, but it's not 'cause it's reusable for every eval I work on. So like you'll see at the end, I'll share the notebook I use now. I basically created like an eval investigation notebook that I use anytime I'm working on an eval. So the other thing that came up here is like this middle one is actually the reason why it failed so much is 'cause I had a bug in one of my helper functions. And so that was also interesting to see like how this data really exposes where like something you're doing may not be working the way you expect. And then I did score these evals. So these are my, also in my notebook, I basically asked ChatGPT to write me a Python function that would compare my human labels to my evals. This was my very first run. This was before I did any sort of iterations to align it. It's pretty hit or miss at this point. And I don't even— I can't even— this was so long ago enough, I can't even tell you what this blank line was. I think it's that I had human labels that I don't have evals for yet, but I'm not totally sure. So up until this point, this was that Sunday through Thursday. I did all of this in 4 days while taking the class, and I was really surprised by how far I could get. And a lot of this was because I just leaned on ChatGPT as a thought partner. Really heavily. Okay, so now a week in, I'm like, okay, I guess I'm a notebook Python developer. Let's keep going. And here's what this did for me. I, oh, I think I already talked through some of this. Sorry. Let's see if there's anything new. Really what this did for me is these two error cases were my most concerning and it really helped me size the problem. Like how big of a problem is this? And that was really reassuring cuz like I know when I can measure something I can improve it. It helped me find bugs. I already talked about that and it helped me get further clarity on my rubric. That was this setting the scene one. So if I go back to this question, I was like, okay, now I have a really structured way to ask, are my changes having an impact? And literally for every model change, prompt change, temperature change, chunking strategy change, or now I've gotten into workflow changes. I have a fast feedback loop. I basically make the change, I run my evals on my dev set that's based on production, and I run my evals on my dev set based on my change. And I literally look at this table and it's never like totally better or totally worse. It's always these evals got a little better, this one got a little worse, and I have to make a judgment call about is this better for and should I release it or not? And I cannot emphasize this enough, like as someone who teaches fast feedback loops, this just felt so magical. I was like, oh, I can actually measure a non-deterministic product. This is awesome. Okay, so what does it look like today? Today I work outta VS Code. The reason why I switched to VS Code is I am a heavy Cloud Code user. Like, I treasure my opus minutes with Claude Code because it designs everything for me. I wanna clarify this because for the non-engineers in the room, I am not vibe coding with Claude Code. Like I'm not letting Claude Code just produce a whole bunch of code that I don't look at. That terrifies me. I still have this philosophy, if I'm going to execute the code, I better understand it. So I pair program with Claude Code. We talk about what's the best way to do this and how should I implement this? And I don't like that pattern. What are the other patterns? And so that's how I get a lot of work done. And sure, Claude Code might write most of my code, but it's really important to me that I understand all of it. And we'll talk about why in just a second. But basically the way that I work is I, you can see here, I don't know if it's very visible, but like, I have a whole evals directory in my project. I run experiments, so I have different directories for different experiments that I'm running. I had Claude code, right, my A/B test runner where it's basically running all my traces against what's in production versus my change. I keep track of, I have evals for that cross dimensions, I have evals for individual dimensions. So this is growing into a gnarly beast. But I love it. What you're seeing here on the right is my notebook. This is my current notebook that I work out of. And actually this screenshot, this in this notebook is this little tool where I can actually select a transcript and it shows me what my helper functions extracted and then what each of the evals returned. And so this is like my magical diagnostic, like what went wrong with this trace tool, which the fact that I can just do this all on a notebook still blows my mind.

The level of joy that I have in this talk keeps increasing exponentially as you keep talking. It's quite fascinating. And so the thing that is really great here is like, not only do you use notebooks for data analysis, you created your own widgets and little user interfaces in the notebooks, which is quite an advanced thing. I want people to know you can do that.

Okay, so I didn't know I could do that. Claude Code did that for me. Like here's what I'm good at, I'm really good at describing what I want, right? It's okay, I see this error in this trace, but you didn't tell me the trace number, like how am I supposed to go investigate? They'll be like, oh, let me add the trace number. I'm like, okay, and I have the trace number, but I don't wanna go to Airtable to look it up. And I have a cache with helper functions and you let me look at the cache, but just put 'em all in one place. So I just demand things from Cloud Code and we work on it together. And it actually, if there's anything you take away from this talk, it's that you don't have to know how to do any of this because LLMs are really good guides. But I have vibe coded too. And what I'll tell you is just make sure you understand what it does because every time I give Claude Code a little bit of a longer leash, things go horribly wrong. So like I'm, I am pair programming with Claude Code. Claude Code is not just generating. A bunch of code with me, for me.

That's great, because I think a lot of people, they try to eat too much, like they try to parallel and like eval everything without looking at the data. You had a sense of, okay, let me kind of work with AI to understand the problem deeply, not let it just look at my traces and do my evals for me, which is like what a lot of people try to do and it doesn't work.

Yeah, I think something about this class like resonated with me so much at like my core that I just wanted to learn it really well. And I think also it's why I didn't wanna use an eval tool. Like I figured if I rolled my own, I would know it inside and out. And even now, like even on the other side of this, I've been watching the eval videos from this second cohort and I'm still like, uh, it just looks more complicated than what I have and what I have is working. So, I'm gonna stick with it. The other thing that's different today is my, my notebook has graduated. So before in my first notebook, all my eval code and my data analysis tools were in one notebook. Now my evals are part of my like normal repo because like I run some of them as guardrails in production. I run 'em on a percentage of traces. And so now my notebook is just a data analysis tool and it, it can execute my evals, but it's executing my production code. So they're no longer commingled. And so in my notebook I have some tools that allow me, the first thing it does is it allows me to run all my helper functions and cache them. So those are my expensive calls. I can do that for my dev set once. And then 'cause my dev set isn't changing very often, The only time I ever have to rerun those is if I change a helper function, and that's been huge. And then I can rerun specific evals, I can rerun all my evals right from my notebook. I do my analysis and visualization right from my notebook. So this is a run I did yesterday. This is how I'm currently doing on that general, suggest a general question. You can see I've gotten this error. It's almost nonexistent. And actually when it shows up, it's showing up because there's some ambiguity. Like the participant starts by asking a specific question and then falls back to a general question. But literally having access to this data and these visualizations has allowed me to iteratively, like dramatically reduce the errors in my interview coach. And then, the notebook also has like diagnostic tools. Okay, what's going on with this trace that failed? Can I go look at why it failed? Is it a problem with one of the helper functions? Is it a problem with the eval itself? Is it a problem actually with the trace and it's a real failure? And I can do all that right in the notebook. The other thing I can do in the notebook, which I don't have screenshots of, is partly 'cause one piece of it is currently broken that I need to fix, but it can pull in my annotations and then compare my annotations to my evals and then score any of my LLMs as judges. Just, I just changed my annotation output. And so that part of the notebook is currently broken. I need to fix it, but it's really nice having all of this in one place.

And again, I just want to say these notebooks are so well organized. They have sections, they have documentation, and that's not just aesthetics. This actually helps when you're using AI.

Okay, so I will say Claude Code created this notebook for me, and because Claude Code created it and like I had to work my way through understanding it, I would regularly update the markdown because I just didn't trust that since I didn't write it that I would remember it. So like a lot of the documentation is for me because like I come back to the notebook and I'm like, which cells do I have to run before I can run an eval, or what's going on with the cache? And yeah, I've learned like one of my biggest takeaways working with, I think I've been using Claude Code for 3 weeks, maybe 4 weeks. And one of my favorite things about it is to be like, okay, let's create a README file. Okay, what did you learn from this conversation? Let's document that. Because everybody who codes knows like one of the hardest pieces is keeping all that context in your head and getting back up to speed. So I'm using LLMs to help solve that problem for me.

Yeah, it's really powerful and it's unique. It's a unique aspect of notebooks that you can have this kind of in, this kind of in-context learning where you can have prose in code interleaved in this way. I find that it really can really help language models, especially since you're updating it with your understanding and knowledge. You're not just like letting it— you're actually like grooming this very carefully.

Yeah, that's a great way to put it, and I think that's really important to me. I gave this talk yesterday and I realized everybody thought I was just vibe coding, and I was like, oh, I need to distinguish this because I don't want to send people off to just create a notebook. If I let Claude just create this on its own, like, the data analysis would be wrong, especially if it was sonnet. Like, sonnet is sloppy. And so actually Andrew Karpafi in his Software 3.0 talk had a really great line that resonated with me, which was, I optimize for 4-line diffs, not 1,000-line diffs. And that like encapsulates perfectly how I work with Claude Code, 'cause I'm really terrified of ending up with a product that I can't maintain myself. Okay, so the other thing that has changed, I now had Claude Code build me a custom annotation tool. You can see here it follows everything you learned in this class. I got a progress bar at the top. This is my trace. It's really nicely color coded and like visual for helping me understand like what section I'm in. I've got an annotation box, I got failure modes, I've got next and back. Everything is key coded so I can do fast keyboard shortcuts. I actually have about half a dozen custom annotation tools and 'cause sometimes I'm annotating the whole trace, Sometimes I'm annotating a single question, so I'm like loading a whole bunch of questions into the tool. And just anytime I'm working on an eval and like my annotation needs change, I literally just have Claude code. Like I tell it, I like the interface from this annotation tool, but now I have new input or a new output format, just spin up another tool and it does it in 2 minutes. Like it's just so fast. So these, I vibe code. I can't even tell you how this is implemented. I just test the inputs and the outputs and make sure that it's working. And then here's the thing that's really exciting to me is that I started this project trying to create a teaching tool and I was using it in my classes. And like my classes, our cohorts are 20 to 50 students. It's not high volume. If something went wrong, I would get an email and I would just rerun it. But over the course of this project, I started talking to this company Vistalead Vistalee is a software product for product discovery. They have a whole flow for customer interviews and helping you synthesize your customer interviews and create opportunity solution trees. And actually, we're in the process of rolling out the Interview Coach as a beta feature inside Vistalee. And what's crazy to me about this, I'm gonna tell you, 2 weeks ago, I had never worked in an IDE. 2 weeks ago, I never had a dev environment and a prod environment. And suddenly, like, I have to make production quality code. So for the last 2 weeks, I've been working nonstop, like, learning how to be a real software engineer. I've been working with Opus and O3 on, okay, walk me through error handling, beat this up, like, where's it gonna fail? And I'm now pretty confident that I have a pretty good production product. We are rolling out to beta customers this week. We'll see how it goes, but I'm crossing my fingers that I've got a real AI app that's doing pretty darn well.

This is amazing.

And as a product person, designer, kind of half-baked engineer, it feels really good to release like a real product where I literally did every part. It's very empowering, and I don't— I could not have done it without LLMs.

This might be the best eval products in production that I've seen. Like, from now on when someone asks me, hey, like, what's a good example of a really, like, good eval, like, a product in production, I'm, this is it.

There's actually, there's one story that just makes me giggle when I think about it. Like, I'm so happy that I'm at this point. So, a few weeks ago, I ran two cohorts of continuous interviewing in July, which means I was getting way more traces than I'd ever had. And after week 1, I started to see a new error. And the error was excerpts would get repeated across dimensions. So like in the opening with the story-based, or in setting the scene, the coach would be like, "This is a great setting the scene question." And then in building the timeline, it would be the exact same excerpt. And it would be like, "This is a terrible building the timeline question." That's really confusing for students. And it's one of the first times I saw an error mode that like expand LLM calls. And so at first I was like, oh, how am I going to fix this? And my first idea was like, I need a question classifier, and then I'll only send each dimension the questions that are relevant to it. And I spent like a day and a half working on a question classifier, which sucks. Like, it's so ambiguous and so hard, and I don't have lots of questions. And like, it was just getting gnarly. And I took a break, and I was at dinner, I was talking to my husband about it, and I was like, oh, What if I just sent any subsequent call the list of excerpts that were already used and told it not to use them? Okay, I went from having an error mode that showed up in 81% of transcripts to having an error mode that only occurred in 3% of transcripts. And what was super fun about this is that I didn't even know how to detect the error because the excerpts weren't exactly the same. So first I had to figure out like, how do I write an eval that even detects this error? And then I had to figure out, okay, this isn't a prompt change, this is actually a workflow change. So how do I A/B test a workflow change? And then I had, and it all worked. And like, I did it all in 3 days. That's when I was like, okay, I feel like an AI engineer now.

Oh, you are definitely an AI engineer. You are in the very center of AI engineer. There is probably, you're like the example that we need to show everybody as an AI engineer, to be honest.

I really love that you're always measuring things before you do the improvement. There's so few people out there who actually take the effort and the initiative to write the automated evaluator or whatever it is, make sure they have measured 81%, then implement the fix and then say 3%. I don't know.

So much of this is 'cause like I teach, just teach this exact same stuff in a different domain. Right? Like in discovery, we talk about start with an outcome. What are you trying to achieve? If you don't know what that is and you can't measure it, don't stop, don't start changing things. So there were so many analogies, like even Hamel's hammering on look at your data. My equivalent of that is go talk to your customer, right? There's just so many analogies that I think I was really had a, a strong mental model to, to layer all this stuff onto.

That's a really good segue into this question of where do you think the product manager and the engineer and whatever, how will they work together? It's very impressive, like, what you've done, honestly, like, to synthesize this together. You've taken a really good mindset, this whole thing. Do you think it may collapse into Cool. Could a single person gonna be doing this going forward? Do you have an idea how, how you might teach product managers this, for example? Will you teach them how to code a little bit? Like, just curious, like, what you're thinking is on this.

Yeah, I wish I had a good answer. I've been really thinking hard about this. I'm a teacher by nature, and so I'm starting to think about, okay, product teams are clearly gonna have to do a lot of this. And if they want it to be good, what does that look like? Who is doing what? And I have some like half-baked ideas 'cause I have been thinking about it a lot, but I don't have a clear picture yet. One of my half-baked ideas is I just separated my eval code from my data analysis. And part of the reason why I did that was like to not reuse code, I, to be able to reuse code. But one of the benefits of that is that like now someone can dig in and do data analysis without having to see the code. And this is really important to me because I wanna create like 20 of these teaching tools. So I have 7 different classes and this is just one of them. And I like have so many ideas for teaching tools, but I can't do this iterative work on a lot of tools personally myself. So like I wanna engage my instructors in helping with these. And so maybe I'm the engineer, but the instructor does the analysis and the evolution. I don't know. And so one idea I have is maybe notebooks can help because I think notebooks are really good at, you can obfuscate some of the code but still let people execute it from a notebook. They can get visualizations. I think this is a new skill that we're all gonna have to learn around. Okay, I wanted to create an interview coach. I had no idea how to do it on day one, but I just took a first step and I tried something. and then I got stuck and I learned how to use an LLM to get unstuck. And then I followed that pattern like 4,000 times over the last 4 months. And I think it's this skill of like, how do you identify the smallest piece that you need to learn? And then how do you work with the LLM to learn that and then iterate from there? Because like yesterday I gave this exact same talk and someone was like, okay, if we were where you were 4 months ago, how should we get started? And I'm like, I just, told you, I told you how I got started. And so clearly there's like an ability there of like, how do you be a self-directed learner with support from an LLM? And I think we're gonna have to learn how to do that even as a cross-functional team, right? So like the product manager has to learn some bits, but then the engineer needs to learn where, yeah, you could write an eval for this, but maybe someone with domain expertise would write a better eval for this. And what does that look like?

One thing that really strikes me is your level of agency is super high and the amount of self-help you avail yourself of proactively with an LLM is quite good. And I feel like a lot of people fail there.

Yeah.

Do you have sense? Is that something you could teach or is it? What— how do you teach that?

I don't know. I've actually been thinking about this, and just yesterday I started thinking about it a lot because a lot of the questions I got yesterday were around like, yeah, but what did you say to ChatGPT? I'm like, okay, forget everything you know about prompt engineering, it doesn't matter. I literally said to ChatGPT— I'm not joking, this is like where I started— I want to learn how to use a Jupyter Notebook, I'm a total beginner, what do I do? Oh, I'm on a Mac. So I didn't even have a development environment set up on my Mac when I started, right? Like I was a total beginner. But the thing is, like, you can ask an LLM all your dumb questions 'cause it doesn't care. And like, I remember when I was in college and I took computer science classes, I would get stuck and it was so frustrating. I gotta go find a TA or I gotta go ask another stupid smart Stanford student that I'm gonna be embarrassed asking my stupid question to. And like, I never did it. I just turned in homework that didn't work cuz like I was too embarrassed to ask what I didn't understand. And so I was a terrible programmer in college. I was great at CS theory, which is how I graduated. But like, I was really bad at programming because I didn't know how to get unstuck. Whereas like now LLMs, like they don't, you can ask a really dumb question and they don't care. So I just, and I think when you're in a conversational flow, like when you're in the the web app of Claude AI or ChatGPT.com, like prompt engineering does not matter because you can iterate, right? So you can just be like, here's my half-baked question, and then it comes back and you're like, no, I don't understand a thing you said. And I literally will copy and paste a line and be like, what do you mean by this? And so some of it is just patience to pick through it and know that you can figure it out with enough time. I went through this past week, so we're like at the 11th hour about to release the Interview Coach inside Vistalead, and I was running through all my like integration tests and one of my integration tests caught an error in one of my Step Functions that was not caught and it didn't catch the error, it just, the test failed and there was an uncaught error and I was like, wow, I've been beating up my error handling, like how do I still have an uncaught error? And it was 'cause I didn't know that this is gonna get really technical for just a second. I didn't know that like JSON path errors in a Step Function are not catchable and they're not retryable, like they just fail the Step Function. And I had written my sub Step Functions to return an error in the payload and like you can't do that. Like it has to throw an exception for the parent Step Function to call it. So I realized like at the 11th hour, all of my error handling between my Step Function, the Step Function calls were wrong. Okay, if you didn't understand all of that, any of that, all you have to know is that this was a very embarrassing last minute thing that made me realize my really reliable app was not very reliable. And this was like last Tuesday and I panicked. And then I was like, okay, you know how to do this, just go talk to Claude and ChatGPT. And I like gave Opus and O3 like, here's my testing pattern, what are the existing patterns, what are the best practices, how do I fix it? And we talked through 3 or 4 different error handling patterns between Step Functions, and I was like, tell me about the pros and cons of each, and we picked one. And then I was like, okay, now look at my code base and how many changes do we have to make and how do we make a project plan? And I do that in planning mode, like I don't want Claude touching my code while we're debating how to do something. And then we write a project plan that's really detailed that I've reviewed, And then we literally step by step implement it. And between Wednesday and Friday, I re-architected all of the error handling in the interview coach and released it on Monday morning. And then ChatGPT 5 came out and I literally started asking ChatGPT 5, beat up my error handling, and it's like, wow, you have really robust error handling. And I was like, yes.

Just so people can play with this, like the Vistalee thing. So that is It's available right now. Can you go sign up for this service?

And— ah, okay. So part of the reason why I partnered with Vistalead is the part of the story I didn't share is that when I launched this tool in my course, I was expecting students to submit their practice interviews, and they did that, but they also started submitting their real customer interviews. And this scary because like I just put this stuff in Airtable. Like, I'm not really— I'm certainly not SOC 2 compliant. And like, people were sending sensitive customer data to me, and I was like, oh, I don't know how to handle that. And I started to look into it, and I was like, yep, not doing that. And so what we get with the partnership with Vistalead is this is a tool where product teams are already storing their customer interviews. They are SOC 2 compliant. They follow all the like data management laws. And so unfortunately, there's only two ways to try this right now. One is through our continuous— excuse me, our continuous interviewing course, or through Vistalead. So if you're looking for a product discovery tool, I do strongly recommend Vistalead. But I am starting to think through what's a light version of the coach that I can make freely available. And actually it's based on those very two evals. Can I at least give people feedback for free on did they ask leading questions and general questions? So I'll have more that's like publicly available to like at least play with probably in the next couple weeks.

That's great. And what's the best way for people to follow you or keep up with what you're doing? What do you recommend that people do?

Yeah, so I blog at producttalk.org. I'm on my, I'm on LinkedIn and X. LinkedIn lately has been my primary social network, but I do check in on X every day. That's probably the best places. And then I'll be in the Discord. So if there's lots of questions, I'm happy. If we have time, I'm happy to take questions now. And I'm also happy to take questions in the Discord.

Okay. Yeah, we could take some questions now. Do you have a little bit of time, Teresa?

Yeah.

Okay.

Robert has a good question. How are you thinking about capturing annotations from end users directly? Is that something that you've thought about? Or do you feel like I am the PM or I'm like building this product and I wanna control all the annotation?

Yeah, Robert, I, you had, we actually had a Twitter discussion about this and I forgot I was gonna tackle this. One of the things I'm really excited about is what's the customer feedback loop in all of this? Cuz that's my jam. and there's a few places of things in the class that I learned that I really wanna experiment with. So one is you guys learned to generate synthetic data by defining like dimensions. I actually think those dimensions should come from customer needs and customer pain points. So like the recipe bot, like if you've talked to people about how they plan their meals and you know what their use cases are, like it gives you way better input for what those dimensions should be and like what types of variations you're gonna have ahead of time. So that's a really clear area where I can see a connection between traditional discovery work and like the very beginning phases of like, how do we generate representative synthetic data? Another area is like almost every AI product has like a thumbs up, thumbs down, was this helpful? And like I did build that into the Interview Coach in Vistalee. I'm still trying to figure out how to do it in our course 'cause they get it by email. But I would love it if we could find, I wanna really experiment with UX patterns of like, how can we get the customer to communicate more granular error feedback? So if they're giving me feedback on the whole response, this was helpful or not, that's great. It like, it tells me to go investigate that trace. If they gimme a verbatim of why, that's even more helpful. But I would love to get to the point of, No, this excerpt, the tip on this excerpt is wrong. Or, and just like, what, like ultimately our product works if it works for the customer. And so as a product manager, you can be a proxy for that. But is there a way, I'm always looking for like, how do we get the, the customer closer to the problem? I don't, Robert, I don't have answers for you yet. Like I've been so heads down building, getting ready for this Vistalee beta. That I've just been an engineer, but I'm going to put my product management, product discovery hat back on and start to run some of those experiments too.

You have some background in computer science education. You did some JavaScript development early in your career. Do you think you would have been just as successful without that? Do people, people looking at this talk, should they feel like they can do what you did without that education?

Yeah, okay, so for like decades there's been this debate in the product management world about should product managers learn how to code. I, I'm really glad I learned how to code in college a little bit. Even as a kid I coded in BASIC on an Apple IIe, but like I really learned how to code in college. I, because one of the skills I got from learning how to code is this like problem solving, debugging, deconstructing like a big problem into smaller parts. And then also like this attention to detail that I don't think you get in anything else. And so I think there's skills you learn from coding that transfer to other domains. And I think coding is one of the best ways to learn those skills. So there's part of me that's like, everybody, I think maybe coding, even with vibecoding being a thing, might be a proficiency, like literacy. I just feel like it teaches such good problem-solving skills. Could I have done this if I didn't have that background? Okay, so my undergraduate was Symbolic Systems. So for people that are familiar with this at Stanford, it's a cross-disciplinary, it's a multidisciplinary program. It focuses on how humans and machines process information. It's also what's happening in the world with generative AI is so cool. In the context of symbolic systems. And I had to take classes on, from computer science, the analytical side of linguistics, so like the underlying structure of language, analytical philosophy, so like mind-brain philosophy, and then cognitive psychology. So it's really like, how do we understand information processing from 4 different domains? What I really took away from that program was critical thinking and problem solving. And like, ideally this is what every program is teaching. But I think when you see how 4 different domains solve very similar problems, you start to see this pattern. For me anyway, I started to see this pattern that I would call a more like problem-solving, critical thinking pattern. And so I think I could do this. I think that experience helped me be good at these types of things. Like, I think what I was good at in this whole process was just like the critical thinking part of it. Is coding the only way to learn critical thinking? I'm sure there's plenty of people listening right now that are good critical thinkers and good problem solvers and have never written a line of code, but I think it's a really good way to learn it.

And do you think— and this may be hard to predict, and I asked this before, but I just want to revisit it— if you were to teach try to teach us the product managers? Is your sense you may introduce some code or teach them how to code since you do think it's the one of the best you mentioned, hey, yeah, this is like the best thing, or would you try to go out a different way or you haven't decided yet?

Just, I'm, this is actually a genuinely hard problem for me. I'm trying to decide what I can uniquely offer. So like the AI race right now is insane. I like, Aman Khan is doing really cool work in his last cohort of his Maven class. He had 30 people using, like, non-technical people using Cursor, which is awesome, right? Is that the right path? I don't know. I think right now it is. Is it gonna be the right path in the long run? Like, I'm working out of a notebook and I love notebooks, but notebooks are pretty technical. Am I really gonna teach all my instructors how to set up a development environment on their Mac? And if they're on Windows, I can't help 'em. Like there, there's some real challenges with the way that I'm working for transferring it to other people. And like in some ways, like wish I was still on a cross-functional team 'cause like I kind of wanna partner with the engineer and play with some of these ideas and what do those handoffs look like? One thing I am gonna do, I'll give you guys all a sneak peek. I'm in the pro— okay. I'm in the process of recording some podcast episodes. I'm gonna be relaunching a new podcast in September. And I'm interviewing product cross-functional product teams about how they're building their AI products. So it's not the like How AI, Clairevo podcast and how people are using AI in their day-to-day workflows. It's here's the story. It's what I did today. Here's the story of an AI product. Here's how it came to fruition. And what I'm really interested in covering in those stories is who did what and how do we collaborate? Because I think this is a really open question. And I don't, I'm not sure what to teach yet, but I do wanna help. Like I do, as an educator, I will be doing something in this space. I'm just still figuring out what the right something is.

I would love to see a visual map of how you went about annotations to the app that you're building. Specifically, there's probably a lot of micro decisions that you had to make along the way. And so Could you unravel some of those?

Were they difficult to make?

How did you know whether something was like a micro decision or something that like required more like you learning something new or figuring out something new?

Yeah, I can talk about some of the kind of key decision points I made and how they came about, and maybe that'll illustrate this a little bit. Yeah. Like one of the big ones was I started— the interview coach was one prompt. Like it was this monster prompt. And the problem I saw with that was that in the coaching responses, like one of my biggest failure modes was the coach would give feedback on one dimension but use criteria from a different dimension. So it was confusing the rules across dimensions. And I think like around that time I learned about workflows and I was like, oh, maybe this needs to be a series of LLM calls. And so I broke it up into 4 LLM calls, but then that introduced some new problems. Like now, like setting this, it's not too much setting the scene, like setting the scene doesn't know about building the timeline and it's literally evaluating the entire interview based on setting the scene. And so like that, that change had a really big impact. Like my overall quality went up when I split up my LLM calls, but my— introduced new error modes. And this is something that like I think is probably really common. There isn't a magical change that makes everything better. It's like a change can improve some things and then maybe introduce some new errors. And I, I then started— I had 4 LLM calls, then I really got stuck on— I even asked this in the Discord. Do I do evals for each LLM call or do I do evals for the whole response? And I actually started by doing 'em one dimension at a time and it got really overwhelming 'cause I was just like, I'm gonna be doing evals for the rest of my life. And I realized like by doing it one dimension at a time, I was like finding a lot of errors, but a lot of them weren't critical errors. And so then I was like, oh, maybe I actually should do evals on the whole trace I should do analysis annotations on the whole trace 'cause then I can still find dimension-specific errors, but I'm prioritizing all of 'em against each other. And that actually made a really big difference. And I still do individual dimension annotations when I'm working on an eval that's specific to that dimension, but I'm doing my error analysis and like failure mode prioritization on the whole trace 'cause that's like what's impacting my customer. And that was a pattern I really struggled with. What's the right way to do this? And I've tried both. I've spent a ton of time annotating things I didn't need to annotate as a result. And I just, I think for me, my conclusion is I should start with the customer experience. The customer gets the full trace. I'm gonna look at the whole trace and it helps me prioritize a little bit better. So that was a key decision point. Choosing to move from Airtable interfaces to my own annotation tool, like I know in class we learned build your own annotation tool and I was like, I did in Airtable, but like really building my own annotation tool, like I still, I'll be annotating, I'll be like 4 traces in and I'll be like, like even this morning I was annotating actually right before this and I was like, why could I add like little pills above my failure mode. So like in this set, once I've used an error mode, it should just be a pill so I don't have to go back and remember what the stupid failure mode was. So like I'm constantly thinking of ways to make annotating faster and I literally just stop what I'm doing and I'm like, Claude, can you add this to this annotation tool? And I'm forcing myself to do that because like you learn in this class, like annotation is so important and if it's not effortless and fast, like you're just not gonna do enough of it. And my annotations are hard, like that coach response is a lot of text. So like my annotations are slow when I'm doing full trace annotations. I have other annotations that like, like when I was doing my question classifier, I actually did my annotations with Claude. I gave Claude a list of questions and I just said, ask me like Claude code. I said, just show me one at a time and let me just tell you the failure mode. and we just ran through like 100 traces in like less than 10 minutes. And I was like, whoa, that was a cool annotation interface that I just used my LLM for. So I think like always just thinking about what's the right tool for the moment. And then I think the other big thing I've learned is just, it sounds so obvious, but take breaks, step away. 'Cause I've definitely had moments where like I'm banging my head at something and I go and have a meal or I go for a walk and I come back and just a way simpler solution comes to mind. And that's like a general problem solving tip. But I, even though I know it, I forget it often. I'm like, I literally am having more fun than I think I've had in my entire career. And I'm obsessed right now and I code all day long. My husband jokes that I broke up with him for Claude. And so like, I really have to rein in Like I have to, I now set a timer on my watch to remind me, just get up and step away. Uh, cuz it's just so fun. It feels like the '90s. It feels like the beginning of the internet and everybody's like geeking out and trying to learn this stuff and everybody's learn, uh, sharing with each other. It just, it feels really similar and really fun.

Yeah. The amount you're having fun is definitely coming through the screen. I can definitely feel it all the way over here. Level of energy, really good. We can keep going on forever, but let's wrap things up. Um, yeah.

And then for folks like in the class, if you guys are on a cross-functional product team and you're building a product, like an AI product, uh, reach out because I'm definitely looking for teams to interview. And you don't have to be an expert. Partly what I want to show with this podcast is how we're all figuring it out. Together, right? I'm not— I've done the guru thing. I'm over it now. I'm just into the let's be a community and share with each other. So that's my goal. It's just how do we share more stories?

That's great. That's, that's really good humility. I would say I'm really impressed. So thanks. I would still call you a guru. Like, it's awesome. Yeah. Thank you so much.

Thank you both. I really enjoyed this class and clearly learned a lot.

Oh, definitely. You definitely learned a lot. This is—

yeah, A+.

Thank you so much.

Thanks.