Building AI Products Right: Hamel Husain on Evals, Data Science & Product Metrics

Welcome Hamill, to the podcast. Welcome to the ABCs for building the Future podcast where we bring on entrepreneurs, researchers, builders of the future, people doing big things with big stories to tell. Hamill, I've had the pleasure of becoming friends with. We've shared marketing, like our marketing journeys now, actually how hard that is versus being a builder. We could talk about that another time. And I took Hamill's course back in May, the very first cohort for the very famous AI Evals course. And I felt like I became a wizard after. And then I kind of helped out in Cohort 2, and I was able to get Hamil here while he's visiting Oakland.

A little bit about myself. I've been working in machine learning and AI for over 25 years. I've worked in a lot of different industries, everything from retail to entertainment, restaurant, you name it. I've been doing machine learning AI in like a lot of different sort of capacities, both as like consulting in consulting, but also in industry. I've worked a lot in like tech companies as well. So I've worked at Airbnb and GitHub and DataRobot, all in machine learning AI roles. I've been working with large language models for a really long time. How long? At GitHub, I was doing a lot of work with large language models for code understanding. And I was involved in research, leading some research around code understanding and published some evals and benchmarks that were used for like the first generation of coding. There's a thing called Codex, like the OG Codex, like way back in the day, not Codec cli, but yeah, like one of the very first models for coding. They use that as a benchmark.

And what year do you run time frame?

I think it was, I want to say it was like 2018, 2019, something like that.

Okay.

I don't remember exact year. Cool. Something around.

So the OG Codex. That's cool.

Yeah, so let's use that. It was called Code Search. Net. It's pretty cool. We're basically. It was a large corpus of code comment pairs. Basically you have like, you know, functions of, you know, various kinds and then like the comments stripped out. And then kind of the benchmark is, could you match the comment to that function? And then, you know, it was kind of de duped and curated carefully so that it was like higher, high quality code. And we can talk about that. But that's cool. It was just basically out of this. It was kind of born out of this other research we were doing around, hey, can we improve search on GitHub because it was like a massive corpus of code and they had some just like latent sort of signals in the code, such as comments. We thought, okay, that would be really interesting to try to see if we can train models that can understand code better.

Very cool. Okay. And then so that was GitHub. And that sounds like you were pioneering on the frontier of these, like what is now becoming, you know, evals.

Like, yeah, I mean, I didn't even think about it as evals. Anytime you train a model of any kind or you think about training a model, you always have to think about, okay, like, what am I going to test it on? Like, what data am I going to have and how do I know if this model is working correctly? You have to think about that, you know, any, anything, like if it's a churn model, forecasting model, you know, class image classification model, some kind of ranking model, you name it. You got to figure out like if you're going to train a model, you got to have some way of measuring with what you're doing is going correctly or not. Yeah, especially like, because, you know, there's a lot of different knobs you can tune when you're training a model. A lot of different choices you can make. So how do you navigate that? Well, you know, having this way to measure things is kind of the central, one of the central activities.

Yeah.

Making sure you're doing that correctly. So.

So it wasn't even, it wasn't even like a word to you. And now it's like evals are the moat. Like everybody's talking about evals and the recent thing. Maybe quick detour. Recent thing is everybody's saying like, what the fuck are evals? Like why is evals? And I saw you made a little like clap back video leading into it. Do you want to talk about just that word evals? Because that's such an interesting word of the time right now.

Yeah. I think evals mean different things to different people. So there's foundation model, benchmarks, so mmlu, human eval, you know, you name it. Meant to sort of measure how smart foundation models are. So there's that and then there's application specific evals. Like, hey, you're developing some kind of coach, like an AI coach for AI health coach. AI health coach, well being, whatever. Okay. It's like you don't want to, you wouldn't use those general purpose benchmarks and be of no use to you.

Yeah.

So how do you measure if your product's working? So it turned out that. So when I kind of fast forward like through a bunch of years. Like, okay, after GitHub, I did a bunch of other things that we can talk about. But yeah, I started doing independent consulting, like helping people build AI products. And the thing that everyone kept getting stuck on is like, hey, I've built a thing. I built a product. It works. I glued together all the pieces. I have basic rag in place. I have, you know, tool calls. It's kind of working, you know, it's, it looks cool, I'm excited. But like, how do we make it work really well? So people were stuck on that and it's the same exact problem. And I was like, you know, to me it was the same conversation over and over again, like, hey, let's look at your data. How are you measuring what's improving? And the same question kept coming back, hey, like, how do you even measure this? Like, you can't measure it. It's stochastic systems. Like, I'm not sure, Hamil, you know, if it's possible to even do that. Like, how can you measure what? Exactly. And so then we realized like, okay, it's really the same thing. And it's a lot of the same principles in sort of machine learning, data science that you would, you would use here. Not exact, you're not doing the exact same activity, but the principles. But yeah, it's like same mindset of, okay, how do you take these stochastic outputs and analyze them and figure out a way of measuring your product and making like taming the beast a bit so that you can have some way of figuring out like, hey, are things working? Or they're not working and having some systematic way to measure that. Got it. And so that's what the word kind of ended up meaning to me is like, hey, applying a data scientist mindset to your product, which means analyzing the data. The when I say data, it means like your product, you know, when. So when you're making LLM calls and you're logging all those traces, like, you know, really analyzing all of that and then like building metrics that help you understand what is happening that are very specific to you.

Yeah.

Again, taking a data science domain specific.

Very problem space specific to that team, building that product.

And the reason why that is even interesting is most people who have, who are building AI products are software engineers, not necessarily data scientists. There is a skill involved in, okay, how do you take a system that has stochastic outputs and how do you measure it properly and how do you tease the right signals out of that data? How do you even go about that? And so soft engineers kind of can struggle with that because that's not something that necessarily taught how to do as a software engineer. And so those. The software engineer thinking kind of breaks down when. If you tried to apply kind of things. Exactly. Into the space of, okay, let me measure my system.

They're used to ground truth. Gotta protect it, ship it, make sure it's protected.

Yeah. It's used to, like, more deterministic tasks.

Yes.

Let's ship it. So on and so forth. So now I would say, like, it's. You can take anything too far. You can. You know, I don't think people should evals max, basically.

Is that a new thing? Eval, Mac min.

Max, basically. Like, so people have asked me, like, oh, should we do eval driven development of some kind? Like, should we write evals first? Should we, like, make evals, like, before we build anything? Oh. And I often say, no, don't do that. Yeah. And it's kind of into. They're like, oh, why? I thought you like evals.

If they're the moat, then.

Yeah.

Yeah.

And I wouldn't even say, like, it is the moat. I would say it is part of your moat. Very critical, potentially. Yeah. But it's not. The answer is nuance. Right. You have to have, like, a product. You want to get carried away in any one dimension, whether it's evals or really anything else. Eval is just, like, an important tool.

Yeah.

That you should know about. You definitely don't want to ignore it.

Yeah.

Because you can get lost in the sauce, like, just ignoring it too.

Yeah.

But so I think that, to answer your question, I think we got a little bit further away from it. Is I think the discourse around evals kind of was this idea of people evals maxing a bit like, okay, like, evals is the most important thing you can possibly do. Like, don't do anything else.

Yeah.

And that's not really. That's not really the case also. There's not one right way to do evals, but there's a lot of wrong ways to do evals.

Tell me about the wrong ways to do evals.

Yeah. There are.

Teams are messing up.

Yeah. So the wrong ways to do evals is try to find a vendor that will just do evals for you, and you just push a button and say, my evals are done. That's what everyone thinks of. Yeah.

Yeah.

Okay. They say, I don't even know what evals are, but I want to, like, figure out my system, you know, if you kind of search around, you'll see all kinds of vendors semi promising to automate your evals. They're kind of backing off now because it doesn't work.

Yeah.

So, you know, people get lost in that and they just like try to say, oh, let me just automate it. Let me have an LLM. Look through all my logs for me and tell me what's wrong. Let me have it. Let me just like use off the shelf metrics. You know, it sounds good on paper. Right. If you're a software engineer, you're like, okay, hallucination score, toxicity score, whatever. And you know, you get a score and you put on a dashboard and you're like, okay, that's. I have a score.

I feel good about it.

I did my evals. Yeah, but that's not the point. The point is not to do your evals. You shouldn't do evals unless you're getting some value out of it. And so it's important. Like if you do an eval, you get some immediate value out of it, otherwise you shouldn't do it.

Yeah. And what is typically the immediate value?

Yeah, up on that. So the first step in doing eval is first understand what is wrong with your system. So doing some data analysis of your system to figure out what is broken. Because there's infinite surface area, let's ground.

It in the health coach use case. So it's like, let's just say I want to lose 20 pounds. I need a health coach that gives me fitness plans, grocery stuff, or around helping make my decisions and protocols to drop that. So like, with that use case in mind, what's immediate value?

Yeah. So let's say you're looking at your health fitness coach and you're looking at the traces and you're seeing user interactions where the health coach is going off track or it's not in interrupting the user before they finished their thought, or it's not gathering enough context, or it's not making the right tool call, or it's not retrieving the right information. There can be any number of errors, but it's always good to look into your specific application in the data and see where are things not working. And it's a. There's a systematic way of doing that. We call it error analysis.

Yeah.

We spent a lot of time drilling it into people error analysis. And it's not complicated. It's as simple. It starts in the base case with counting. Look at your traces, you take some notes about what is not working and you can categorize Those notes and count those notes. Okay. Say like, oh, okay. I looked at 100 traces. I found this issue with tool calling. The specific tool calling of, you know, buy groceries from my list for me like 20, 20 times. And then, you know, that was like the thing that I saw the most. That. So you have some concrete idea. It's not abstract, like hallucination score. Like, you're like, hallucination of what? And like, so there's a way to use a generic metrics, but it's kind of people. We can talk about that later. The idea is that you need to focus on what is actually going wrong and then try to iterate on it. And then. So first you, like, figure out what's wrong and then you decide, like, do you need an eval? So in the case of like, are you. If it's like calling the wrong tool, but for whatever reason, you. In your API call, you provided the wrong tool in the list of tools, and you're like, oh, that's just stupid. Yeah, like, that's my. That's my mistake.

That's my bad.

Like, you don't need an eval. Yeah. I would say just fix it. Because you have pretty high confidence. You should have an intuition. Like, that's just dumb.

Yeah.

Like, I can just fix it and let's move on. Let's. Like, the whole point of evals is to make your product better.

Yeah. The system.

So, like, it's not about doing the eval. It's about is your product getting better as fast as possible.

Yeah.

And so, you know, and. But there you will come to, like, some other errors that you're like, I don't really know how to fix this right now. For example, like, well, I don't really like this fitness advice that's giving my users. It's like, sounds very generic. It's not really personalized enough to them. You know, I don't like how it's the tonality of it. It's a bit, you know, too paternalistic or something. I'm just making that up. Yeah. Yeah. So you know that one you're like, I don't know, like, it's not. Maybe it's like, you know, maybe you have something in the prompt already and like, I'm not really sure how to fix this. That's like a better. That's a more telltale sign that an eval will provide you value because you can start to try to measure that. Yeah. Like, okay, you know what? This problem I have to iterate against now. We try to iterate against something. Now that's when like the eval starting to give you value because every time you change something you're getting a signal that you trust if you build the eval correctly and you trust it now after you put in that work now you can iterate rapidly like, okay, I'm going to try XYZ abc. I'm going to try all these things. Now the nuance here is again the difference between the first example and the second example is the first one is very obvious, like you could fix that trivially. The second one was I'm not sure there's a spectrum and it kind of makes you have to make a judgment call.

Yeah.

Of like, hey, when do you fix shit? And when you. Not when you don't. Typically when you're starting out in this mode, you won't write evals for a while because you. In your list of things you'll find a lot of low hanging fruits like, oh, that's dumb. Like actually have a syntax error in my code when you're doing retrieval when this happens. And like that is nothing. Like I don't. I'm just going to fix it.

Yeah.

And you just go fix it. Yeah. So, and that's counterintuitive, right? As a software engineer, like oh, I want like to write the equate eval exactly to unit test. And they say like, okay, I'm going to write a test for all of this. And you should write tests, you should write your software test for sure. Like, and that's always going to be there. Doesn't matter if it's AI or not AI. But specifically I'm talking about AI evals is like the most, the more, a little bit more expensive kind where you're trying to, you know, and there's different flavors of AI evals.

Tell me about the flavors of AI evals. And then cheap to expensive. Like I don't know if there's some sort of like spectrum there for people to grasp.

So like there's different kinds of AI evals. One is like code based eval where you can make an assertion like does the string, does a string contain a user id? You know, you can use a regular expression or something to test that and you don't need, it's not that expensive. You can just run it like a unit test. And then there's LLM as a judge, which is like, okay, now it's something more subjective, like not the right tone, being too paternalistic, etc. You might need an LLM judge, but sometimes you don't Sometimes you can get a little bit savvy with it. You're like, I'm gonna, I'm gonna test for these keywords that are sort of high overlap with this problem.

Yeah.

Or something like that. You can be crafty, but just, you know, that's more advanced. You know, you, there's a spectrum again. And so, you know, the LMS judge is more expensive. It takes, you have to make an LM call and you have to do an eval of the eval. So you have to say like, okay, because if you're going to use an LLM to judge something, you have to know, is it judging correctly? So you have to go through a process of labeling some examples and figuring out, oh, this is the LLM judge. Do I agree with it? Yeah.

So real quick, let's say I'm CEO of AI startup and I got 30 minutes, 60 minutes this week with my team to solve AI evals. What the hell do I do? Where do I start?

If you have 30 minutes, you should start, you should definitely start with error analysis. You should start with basic, you know, taking notes on traces in the specific way that we teach. We teach you like, okay, how do you start looking at notes or traces, taking notes of them, categorizing those and just counting them.

That's it.

And that's it. If you do that, it'll, it'll motivate all the activities.

What does that mean, motivate all the activities?

It'll motivate like further activities like evals. But you want to be motivated to do evals like it should be. It should be apparent to you that it's necessary. So you're going to find a lot of things by looking at data. You're going to find a lot of value in it, even if you do it for 30 minutes as a team. And then you'll quickly, first of all, you'll get a lot of to do's from that. Yeah, you can go stop and then go do a bunch of engineering, fix your product. But then you know, you might decide like, okay, this is like, you'll find an error that is like difficult enough, that's important enough. You're like, hm, you know what, I think the eval would be really good here.

Okay.

And depending on the cost of the eval, again, if it's code based eval, you'd be like, okay, you know what, I don't want user IDs in my outputs. Or you know, I want to make sure, like if I have code blocks in my outputs, you know, they need to have like a code fence that is, you know, looks a certain way or whatever. You know, you can have like these cheap evals. You can have the more expensive ones. You have to make a judgment call on like if it, you know, if you need it or not or whatever. But you should start with the data analysis. So a lot of people leave out the data analysis of evals. The kind of the debate on Twitter is like, are evals necessary?

Yeah. Wait, okay, so on this one there, it seems to be. There's so many fucking hot takes out there right now on Twitter, especially the last two, three weeks where people are like, anthropic doesn't use evals. What then what, like, what are we doing with the evals are dumb or whatever.

What.

Tell me what your. What's your reaction to that?

Yeah, so there's a couple of problems. There's a lot of things going on here. One is some of the most successful products right now commercially are coding agents. And they were the first. Yeah, copilot, whatever. And the reason why they were first is because that's a very special situation where the domain expert and the developer are the same. So you can collapse the activity of understanding, looking at your data and you can collapse the loop of that pretty kind of far in ways that you can't do with other products. Totally.

Like, I'm not a health coach, I'm building an AI health coach.

Yeah, okay.

Yeah, got it.

But like, you know, if you are using a product in anger all day long and you are the expert in that domain, then you can short circuit some activities. That's one second is with coding agents. They have baked in the evals into the model. So the evals have been done upstream.

At the foundation level?

Yeah, at the foundation model level. I mean, the models, if there's one domain where they have really went hard on is coding, they want to make coding really good. So what they've done is before the model gets to you, they have lots of benchmarks, tests, etc. When they train the model using RL and the RL, the feedback mechanism in RL is giving like a reward signal. It's like, hey, is this change or is this code good or bad? And that that mechanism is an evo. And so that has been trained into the model. So by the time it gets kind of downstream, you don't really need you. You could use a lot less rigor for those evals. So the evals are kind of trained, you know, the signal is like trained into a large extent. And so now. Yeah, it's not as. It's no surprise, but I would say, like, that's. That's another angle of it. And. And then I would say they are doing evals. Even. Even so. And so don't.

Aren't some Open Air people in your class for EVAs?

Oh, yeah, there's a lot of OpenAI people. There's.

How many people?

Think there's almost 50.

Almost 50 open air employees have taken your class. Yeah.

Yeah.

Wow. Okay, so. So wait, that to all the Twitter trolls, like, yo, what?

That's just. So look at the data on Twitter, right? It's like, you know, anything on Twitter is like, I think what's going on last two or three weeks is like rage bait. And this is across a lot of different subjects. Like, you don't need evals. You don't need to code. You don't. Yeah, whatever. You don't need xyz. XYZ is dead.

Engineering is dead. Rags dead, whatever.

Yeah, like that. Like all this stuff doesn't make sense. Right? But it's like. Okay, so I would say the way the anthropic is. So if you read deeply, and not even deeply, but if you just read what they said about this is they, you know, for Codex or. Sorry, for Claude, for cloud code, they dog food the shit out of it. They have their entire engineering teams constantly give feedback in Slack.

The domain experts. Yeah, the domain and the users.

Yeah.

Collapsed in one.

Yeah. And they're constantly giving feedback, and the developers of cloud code are constantly looking at that feedback. And so, you know, that's like constant error analysis and constant looking at real feedback. And they're dogfooding it.

Constant signal.

Yeah, constant signal. Now, okay, so they are doing some level of evals. There's a spectrum and there's no, like, wrong ways. There's no, like, right one, right way to do it. But it's. It's when people think evals, they think of like, hey, here's a suite of tests. And these are evals, which is true. That's what evals are. But it's. You cannot separate evals from data analysis. Hmm. And so wait, what do you.

So you can't separate evals from data analysis. Tell me why.

Yeah, because the data analysis should drive evals, should tell you what to eval, how you should eval. Then this is for product specific, application specific stuff. You know, it's better to have your data analysis, like your problems, like if eval is based on your specific problems in, like, Facebook failures, because that's where the value, the highest value of evals are, is like to help you fix a problem.

Yeah.

And so, you know, doing some analysis of that is helpful. Yeah, right.

Of the failure modes.

Yeah. And I'm pretty sure, like, you know, Anthropic is doing lots of data analysis. I know they are. They have some blog posts where they're taking all the conversations people are having with Claude, they're clustering them in various ways and they're looking at what people are doing and then obviously they're looking into different failures and how they can address it.

On that note, because you said that in this case its developer is the domain expert, so it's kind of collapsed, so the feedback loop is way tighter. Literally could be the same person. Right.

Clog code. It is.

They're dogfooding, so it's the same person.

Now the danger is people hear, oh, like they're dog fooding. You don't eat your. Maybe they don't need evals. People then want to jump and say, oh, we're dog food.

I'm gonna dog food my own AI health coach.

But everyone thinks they are dog fooding. Ah, but are you, Are you dog fooding on that level? Yeah, like, are you dog fooding on that level? That where you are the expert. You are using it in anger all the time. Yeah. Because like, if you're.

What does in anger mean?

Oh, so basically, like you're actually using it to get something done in real life. You're just not using it for the sake of testing it. Of like testing in an abstract. Like you actually need to make.

So its output affects your livelihood.

Yeah, okay. Yeah, yeah. So like the people using it, working at Anthropic, they're trying to ship code in real life to production and they're like, hey, I want to use this. Like, like I want it to work. Yeah. And if it doesn't work, it's going to irritate the hell out of you and it's going to slow you down, it's going to make your life worse. If it doesn't work, you want it to work. Oh, so like, you know that's real. Like, you can't, can't fake that. That's. That's real dog food. A lot of people, that's a different level of dog fooding. So like, when people say dog fooding, everyone likes to think they're doing dog foods. Like, okay, are you really, like, to what extent are you doing it?

Yeah.

And so, yeah, if you're doing it like that, then yeah, got it. That's powerful. Like, I mean, that's always going to be powerful for any product, AI or not.

Yeah.

If you can find the situation where you're doing that as close to the.

Real world use case and you're dependent on the output for the value.

Right. Like, yeah.

Okay, so that's real dogfooding. Have you seen cases where it's like the opposite of real dogfooding and a team is like, they think they're dogfooding. Could you describe any, you know, maybe in the abstract the case where that happened?

Yeah, I was. So I was helping a team just last week, and it was a platform that sends emails to. Is like recruiters send emails to potential candidates to about jobs. And, you know, they're showing me their traces. And we looked at some. We just like two minutes in, we started looking at traces and I was talking to a data scientist, kind of like me. Your person.

Yeah.

And the traces that we saw, they were worded like, based on your background. Based on your background at X, Y and Z.

Epistemic me.

Epistemic me. Based on your experience. Epistemic me. Blah, blah. And I said, that sounds like just very generic recruiter speak. I hate. Don't you hate this? Yeah, that's what I asked. I was like, if I get email like this, like, I'm gonna just delete this. Yeah.

Archive right away.

Sounds like so generic. And I get emails. I've gotten emails like this, like, for so long and I can tell, like, there's put no thought into it.

Yeah.

And they're like, oh, okay. It's like, that's a, that's an interesting point.

Yeah.

Because like, when they showed me the data, they're like, oh, like, everything works.

Yeah.

And then I saw, I was like, I don't really think this is working.

Yeah.

That was my, like, impression.

Yeah.

And so now I'm not the expert, you know. But are they using it to, like, does that mean get a job? Because they're using it to try to recruit somebody. Yeah. Like, some of the users are, but, like, you know, I'm skeptical. Like, the loop is completely closed. Right. It's like, if you are sitting there, like, you know what? I really need to, like, recruit some people. I need to get their attention.

Yeah.

I would suspect that that email would not be written that way. Like, they, they would have figured. They were like, hey, stop. Yeah.

Like, my whole goal is to fill this role.

Yeah.

Or I will get fired. That is what I'm getting paid for.

Yeah. Like so that's like you would iterate on the prompt more. You'd be like, how do I stand out? Yeah, that'd be different than how do.

I get more conversion rates for this click through rates.

Yeah, yeah. And so that's what happens all the time. You're a developer working on some AI product. You're like, looks good to me, looks reasonable to me. Yeah. And you got to close the loop on like, hey, is this good?

Just take a step back for all the developers out there. What does closing the loop on evals mean?

Yeah, it's basically defining what good means from a product lens really deeply what matters to your users and designing metrics like curating those metrics that matter and iterating really fast on them.

Okay.

So getting to really know your users and yeah, using the data to potentially do that, but also putting your product add on.

Got it. So for the engineering teams that have product managers, designers, researchers, doc people and the engineers, what would you say to them if they're starting from scratch for that problem? Closing the loop.

Yeah, you have to really understand your users. Right. It's like, so you can make an argument like, based on what I said, like one spicy argument. And this is not nothing to do with AI. You could say like, if you're really good or if you're really invested, let's say in the product, like, it doesn't matter if you're an engineer. Like, you should think deeply in a product sense. Like, what are you building right now? Sometimes there's like too much surface area and people just don't get there. But I would say, yeah, I mean, you got to put your product hat on and you have to make sure that it's not just evals for eval sakes. Like, you think about your business objective and like your is your evals. Like, how is your system behavior like helping you achieve that?

Yeah.

And you can try to measure that. You can have product metrics as well.

Yes.

Other than AI metrics, you could say like, okay, for this email thing, you can track like open rate, response rate, whatever, you know.

Yeah.

And you can, you definitely should. So there's a whole spectrum, like, you know, evals. I'm not the authority on what the definition of evals are.

I mean, I would say some people think that way, actually.

I mean, like, no, because like, you know, it can mean a lot of different things. I think in the strictest sense, in the most narrow sense, evals are tests like that you measure AI against. But this just doesn't live in a Silo. So like you have to have data analysis and you have to have, most likely, you have to have product metrics. There's offline tests, there's off online tests. A lot of people, you know, in the strictest sense, like evals are only offline tests. But like you need online tests, you need to measure what's happening in production, how people are reacting to your product. And those metrics can give you some. Another lens, like, oh, okay, why are conversations dropping off at this point in the user journey or whatever. And you can try to debug that and say, okay, I need some offline tests, I didn't know this was happening, or whatever. You need a whole suite of things. The limit devolves into being a data scientist, because this is exactly what a data scientist is. Even before LLMs, a data scientist was someone that helps you understand your product from a data. From like you. From a data perspective. Using all of these tools.

Yeah.

At large, like doing product analytics, data analytics, offline tests, online tests, kind of figuring out where the story is, why things are going, you know, why things are happening. Doing like causal.

Tell me about that. Where, where the story is. Is that something that data scientists, you would say, like right now should be focused on, is the storytelling behind the data for this is that. That caught my attention.

Yeah. So what I mean by story is in a lot of ways data science. The good data science in businesses.

Yeah.

Looks like investigative journalism.

Like so good, good data science in big businesses. Looks like investigative journalism. Tell me more.

Yeah, so if like your, if you're having margin leakage or your churn rate has all of a sudden skyrocketed. Skyrocketed, or you have a leaky funnel of some kind, or customer satisfaction is low or whatever the hell it is. Right. There's a lot of skill in sort of figuring out like teasing out the story of like, huh, this event happened. What can I learn from both qualitative and quantitative analysis of what exactly happened here? And what can I learn? Right. And so what does that sound like? That sounds like if you pick up the newspaper and start reading some breakdown of something, you will often find both qualitative and quantitative analysis of a situation. Like, it'll tell you some story, some anecdotes, some interviews, then often there's some data says like, oh, like if you analyze this thing and this supports that, you know, the qualitative gets more color. Sometimes it actually adds some things they can't explain with data and gives you a holistic understanding of a problem that then you can then use to say, you know, so that's like one part of it. And it's like you communicate it in a way to everyone else that they can understand. I see. And you see, you know, to like, make some kind of decision. Yeah. Now you know, and you can take it further than that. It's like, you can't reduce data science, only to investigate.

That's a good.

But that's like a good. Yeah, sort of. People don't think of it that way. They're like, oh, it's just about like doing statistics and run some code on some data and whatever. No.

Do data stuff.

No, it's like, it's a little more holistic. You got to put your, like, you got to put the whole context of what you're doing in there.

Yeah, that's quite interesting. So I want to zoom out a little bit because we deep dived on the Twitter trolls, we deep dive on the rage baiting what evals means and all that, and how people get could get bootstrapped, which is just go look at your data to start. Go count things. And that will spawn a whole set of activities. And it's up to you as the team to understand your users and the product metrics and the business outcomes that you want to drive. To figure out where do I spend my time and effort as a team with the resources we have to drive those things. Pick what kind of evals, pick what kind of tools. So zooming out, when have you had to be an investigative journalist in your career? I want to hear more about you, and I think other people want to hear more about you too. So you, like, were the pioneer around kind of the first coding use cases in ML for GitHub. Why did you quit corporate America? Like, when did that happen? Like, tell me more about that.

So I just want to say I don't feel like I'm the pioneer of any coding anything. Like, you know, I was part of a team. They're very talented. I was involved in it. You know, I helped push certain things forward. But there's a lot of people that do a lot of things. I'm just one little brick on a big, like, cathedral of people.

Yeah.

I'm standing on shoulders of giants.

Yeah.

So I wouldn't. I don't feel like I pioneered anything. This is like, letting you know.

Okay, okay.

So okay. So yeah, I was working on eva. So. Okay. I was working on AI ML stuff for a really long time. So yes. Like, why did I left? Always find, like, I'm in so many Meetings and they're not meaningful.

Yeah.

You know, performance reviews, whatever kinds of politics that might happen. And I just found it very deeply unsatisfying and always, you know, let's say I'm sort maybe typical, maybe atypical, but, you know, I would switch jobs every two years or so.

Okay.

And GitHub has stayed for like longer than that because there's. We got acquired by Microsoft, so.

Okay.

Strong incentive.

Yeah, yeah, yeah.

But always thought something's wrong with me.

Oh, you always thought something was wrong with you?

Like, oh, you know, maybe it's like, why don't like these jobs? You keep like switching jobs, hoping like whatever, you just like the next one or whatever. Then I realized, you know what, it's not me or it's not something wrong with me. It's just that I just like operating more like an entrepreneur. So like, I want to own the upside. I want to own my own mistakes. I don't want to, you know, if I make a mistake, I want to own it. If I do. You know, I don't want to convince someone that I'm doing a good job. I just want to see the result. Yeah, like, hey, you know, is a client successful or am I able to sell this product or, you know, renew this engagement or whatever it might be? Like, I want to kind of have it be more real.

The upside and the downside, the accountability and the ownership and that would be.

More fulfilling to me. And you know, and so I didn't really know how to do that.

Yeah. Wait, does anybody really know how when they start?

I didn't really know how to do that and thing, you know, just by accident in a way. So like, you know, I was working at. After GitHub, I decided to go work at a small startup just to kind of get my directional sort of movement in the way of, you know, doing my own startup potentially. And then I like left that and I, you know, sort of explore doing my own thing. In the meantime, a company contacted me and they said, like, hey, will you do some like, consulting for us? So I did that. I was like, oh, this is pretty cool. Like, and then I just started doing it for more companies. Like more companies, you know, wanted help building AI products. And so I just kept doing it and yeah, here we are today. Like, yeah, I just enjoyed the autonomy helping people. You know, I told myself, okay, let's do some consulting and see some what problems resonate with me eventually build stuff. And then at some point I kind of noticed that everyone's having the same problem. That I was working with. And that's when I decided, like, okay, I should teach this course and write about it more, because ultimately, like, I want to find a way, like, I don't have to talk about e bels like, can I kind of solve this problem and move on to the next thing?

Yeah.

But everyone that I work with seems blocked on this, so. Okay, let me put something out there that kind of scales beyond one to one, engage, like, engagements that sort of, like, help people do it. And I wasn't convinced that I needed to build tools.

Wait, you weren't convinced you need to be able to build tools?

Why?

Is it because of the nature of the problem that you described, or.

Yeah, I mean, like, the tools are already there. My friends are building tools. And they were like, okay. And like, it wasn't that the tools were lacking as much as. So, like, how people are using the tools.

Methodology.

Yeah. And so I felt like you could build a tool, but it's really like, even if I did build a tool, I could envision people not understanding the right process, and I didn't see that you could just automate the process completely. So I felt that, like, okay, teaching people the right way to do it is gonna make an impact because everyone wants to build tools. I want to build tools too. It is the most scalable way to generate wealth. If you find a way to, like, build a product, it's high leverage. Yeah, high leverage. And so I might do that, but I'm not sure, honestly, like, especially in eval space.

Yeah.

It's extremely competitive.

Okay.

And I agree with Peter Thiel.

Yeah.

Like, you don't like competition is. Is, like, for idiots.

Yeah. Competition is dumb.

Yeah. Don't dislike. I totally agree. And so, like, I would say in the LLM space, evals is extremely competitive.

Got it. Okay. So tell me about your entrepreneurship journey. Like, how many years was that? It sounds like you jumped to the startup and you were immediately contacted, maybe from your reputation in the space to work on AI products, and then you just went from contract to contract to contract. Can you get into the details of that for maybe other builders of the futures and other entrepreneurs who might be like, hey, I kind of don't like politics. Hey, I kind of want to own the upside and downside, but I'm a little scared to jump ship. Tell me more about that. How'd you handle that?

Yeah, so first I started with, like, I took one contract on. I think they paid me, like, $500 an hour. And, you know, it wasn't like, infinite or anything. It was like five hours a week or something. I was like, oh, that's like, more than enough to live on. So I did that. I sort of. I already had a blog at that point, which I wasn't trying to use for any purpose at all. I was just. Had a blog that I've always written, and at that point, I didn't really have a newsletter or anything. So I was like, oh, like I should turn on this convertkit thing and just turned it on. And then I sort of, you know, let people know that I'm doing consulting, you know, on social media or whatever. On my blog.

Yeah, yeah.

And then slowly, like, people started contacting me, like, here and there. Word of mouth, you know, I got more engagements. Sometimes I would do paid engagement. Sometimes I would do, like, free office hours if I thought, hey, like, I want a different flavor of. I want to teach people about a subject. But I feel like there's not enough examples out there. Sometimes I would do free consulting to get the.

Yeah.

To unlock the example that would help people.

I see.

And then, yeah, I just kept doing consulting. I then, like, what happened is so, like, also I was into. At some point, I was into fine tuning. I thought it was fun. At least I don't do it much anymore. But at the time, you know, I had a friend, he was like, I'll do this course with me fine tuning. I was like, whatever. Okay, let's try. I'm not doing anything. You're not doing this consulting. Just whatever. Like, let's. Let's try stuff.

Try stuff.

So last year, I did this course with him, and yeah, it kind of blew up. Like, I got my clients involved to, like, you know, I was working with the replicate, which is a AI company. They gave, like, all my students credits. They gave the students credits. And then, like, hugging face wanted to compete. They gave credits. Then OpenAI wanted to compete. They gave. And then became this crazy, like, thing bonanza. And it was like, wow. Okay. Like, it was like a mini conference, like, by accident. Just like. And then it was like all my, like, invited all my friends became, like, kind of crazy. It was like, oh, okay. And then from an income perspective, that was huge. That was, like, eclipsed all my consulting by, you know, maybe like, 5x, like, whatever. I was making consulting.

Okay.

I was like, okay, that's really interesting.

Like, that's high leverage.

Higher leverage. Yeah.

And impact was higher.

Yeah, impact was higher. And then I was like, okay. Then I didn't do that. I just shut the course down. It's Like, I don't want to do courses. Like, I don't know. I don't even know how this happened.

Yeah, you just kind of. It sounds like you just sort of walked into it after, like. Yeah, okay. It sounds fun.

Yeah, I walked into it. I realized I, like, you know, certain things clicked. I'm like, okay, like, affiliate marketing works and, you know, having, you know, these different, like, whatever. Bonuses for people. Whatever. It was like, works, like, all these things are, like, really interesting. I never had that experience before.

Yeah.

I've always traded my time directly for money. Yeah. Even in the consulting, I was still trading my time for money. Yes. The first time I, like, had any experience selling anything online, it was totally by accident. And, like, people pay me for it, and I was like, oh, my God. Like, yeah, I would, like, go to sleep and then wake up in the morning and then be like, all these sales.

Yeah.

While I was sleeping. So it's totally different. Like, totally different.

How did that feel? To wait to go from I'm trading time for money to, like, I slept overnight and stuff sold on my behalf. How did that feel?

Yeah, that totally rewired my brain.

It totally rewired brain. Tell me more.

So I thought, okay, I can't go back. I can never go back to a job. In fact, I'm not sure exactly if I want to go back to consulting.

Okay.

Like, 100%. I'm like, just tight. Okay. This, like, time for money has to go.

Yeah, yeah, dude. Tell me about it, bro. Tell me about it, bro.

Yeah, so I thought, okay. I mean, it's easy to say that, right? Yeah. Time for my. I didn't understand it. Yeah. Intuitively, like, get it versus create assets. Like a. It was like, yeah, time for money is mad, of course. But, like, I didn't think it was a reality.

Like, you didn't have the experience that.

Yeah, I never lived it, so it was just like a abstract possibility. Yeah. So once it happened by accident, and I worked really hard to like, make it that course. Like, when it would start working, I'd start putting more fuel in the fire or whatever.

Yeah. Yeah.

But, like, whatever it was, I was like, wow, okay, that's super interesting. I was like, okay, well, that's something in my back pocket. Like, I don't plan on doing course. I don't want to just do courses. Like, what the hell? But then I was like, after a year went by, you know, started like, helping people. Not so much with fine tuning, but this evals, like, constant. When we're evals. I'M like, okay, you know. And I started writing about evals. I wasn't. And I'm not like, I'm not gonna do a course or anything. I'm just writing about. Always like make habit of writing about whatever I'm doing.

Clarifies your thinking.

Yeah. So I was like. Kept writing about it. People get like really interested in the topic. And I was like, yeah, I need. Maybe need to do a course, but like I should do a course to like just get the info out there. But I don't really want to do a course again. Like I'm. The idea of a course doesn't sound fun to me on its own. So I was talking with Shreya. Shreya is my co instructor at the evals course. I hate it. Sounds like somebody needs to teach people about this. She was like, yeah. So we decided, let's do this, let's do it again. I like took all the stuff I learned from, like how to sort of, you know, all that logistics experience from last time, decided to do it again. This time it was like way more successful than the last one because I have some experience, but I think, you know, and this time we're able to go a lot further. So I was able to like do. I did. We're doing it three times. So we did like once we reinvested all the money into the second one. Now we're like reinvesting all the money again into the third one. And by reinvesting, I mean like, you know, having production quality videos.

Yeah.

We're also like, we're writing a book.

Yeah.

And then we are, you know, giving people like some tools.

Dude, I saw that. I. I don't know if you could shout that out, but I got an email about the tool that you're giving in this. Do you want to.

Yeah. So tools is like, it's very like, like we're not building. We didn't build anything. We just want to see what people are using and validate if it's helpful. So like there's this thing that. So this thing called Delphi AI. And I know you use Delphi.

Yeah, yeah, yeah.

So you know, you can just. This is like a private GPT.

Yeah.

Where you can put all your information in their digital mind. Yeah. So like, you know, we pushed everything we've ever said about emails, like all, all the lectures, all the Discord chats. So we have that Discord community for the course. We uploaded all the chats. Like we curated it. Not like, yeah, yeah, we didn't like just Raw. Yeah, we kind of curated, like, the questions and answers from that. Yeah. And then all my blog posts, all of Shreya's papers, all of Shreya's talks, all of my talks, all of the office hours, lots of content.

Cool.

And put it into this thing. And then what we want to see is like, okay, do people find this helpful? Just, like, what kind of questions do people ask?

Yeah.

Because we can see what questions people are asking, and then maybe that gives us an idea. Okay, should we build something? We could have built something like that. But we're like, we don't want to necessarily invest in things until we kind of figure out if there's pmf, if potential, there's some kind of value in it.

So wait, quick, funny question. Is it you or Shreya as the avatar on deli, or is it both of you?

No, like, they require avatar.

Yeah, they do.

So I just put myself.

Oh, cool.

They're like, well, you have to pick one avatar.

Yeah.

And I was like, just. I was doing it. So I just.

Okay, so how can people get access to Hamil Delphi?

Oh, they have to get by the course. So it's only for people in the course.

Okay, so the. It sounds like the most trained AI digital mind from all evals related questions, you can only access by going to your next course. So when's your next course?

It starts October 6th. I will say, like, in the. In that AI assistant, I have a disclaimer. I was like, hey, if you want to be. If you want to be rest assured that you're getting the correct answer. And we also have a feeling, like, because I gave it to my friends who are working on evals. Yeah. And they're asking questions of it, like, all the time. They've told me it's helpful. And so we have a feeling, like, maybe there's some shy people out there. Maybe people will just, like, you know, use it for interesting things. Yeah. Yeah. And so we just want to get a sense of, like, what's going on. Like, if people are. Yeah.

Yeah, I could totally use this. That sounds so useful. Okay, wait. That's so cool. Okay, wait. So to summarize this arc, it really sounds like you went from I'm tired of corporate America politics. Like, I just want to have impact and leverage, and I want to own the upside and the downside. I want to have accountability. And then it moved into, all right, well, I kind of bootstrapped my entrepreneurship journey by getting your reputation right and getting contracts, services, contracts, trading time for money and Last year, a few years later. It sounds like after that. Last year was when you realized oh, I could build an asset from my knowledge that sells overnight while I'm sleeping. And it led to okay, now I got a rep doing that, figuring out the logistics, figuring out I can make money from affiliate marketing, figuring out I can make something out of it. To evals course one I would say.

Like, particularly in like tech circles, Silicon Valley, you know people, there's like a, the idea of like selling a course is definitely lower on the totem. Like oh, lower. Like social status status. I see maybe even negative social status.

Negative social status to sell a course in Silicon Valley. Negative social status.

I would say negative social status. Okay. You know it's like building a product like a, you know, some kind of big product that you need VC funding for is like that is the social status. Maybe bootstrapping your own. Yeah, like smaller company that you're building a product is like slightly less like. And then like somewhere, somewhere else is like down, down low is like doing a course of some kind. Then like you know, it's really interesting. But then like having a job is like maybe above that, maybe below it. It's hard to tell.

Yeah.

Depending on what you're doing. There's different, you know, ad hominems people have for like people doing courses and so boys is course boys. But I would say like there's some truth in that. And there's also. And there's like, you know, you can call it like meta creator which is like hey, you know you're not building but you're kind of talking about building. I will say like it's important to have a perspective in here. Like so there's definitely a lot of grift out there in oh courses. Like I mean this is tremendous. It's like with anything, it's like anything dude. You know. So like marketing's a lot of grift courses. Definitely. Like there's a course for everything and there's a lot of course how to make courses. Yeah, there is. Yeah there's a lot of courses on.

How to course from people that haven't had courses that are successful.

Yeah, yeah, yeah, yeah. So like you gotta, I mean but on one hand there's a lot of that and there's a lot of that reputation is deserved in a sense like in some respect. But then like if you think deeply about it, you know, education is something that people that has value. Like you spent a lot of your life like getting educated on stuff and you pay colleges like lots of money, you know, in part for education. Not 100, but like in part, like for the education. And you know, like, okay. And it's like expensive to create education if you do it correctly. So it's not like, if you do it like with the right mindset of like, hey, you actually want to really create value somewhere you're not, you know, and you want to make it sustainable, then like, yeah, it's totally fine. You know, I think it's an interesting kind of. We'll see what happens. Like, I think you gain some skills by doing it.

You know, you gain different skills for sure.

Yeah, you gain some skills. You have to, you know, you gain some marketing skills, you gain communication skills, you know, more writing skills. There's all kinds of tools and things that you might, you know, whatever. It's pretty high leverage. Like marketing skills, like good marketing skills, technical writing skills. Like all that stuff is really high leverage. And I think people, we are told as engineers that to kind of make fun of people that are good at those things in a way.

Really?

I think so. Like, yeah, like if you're like, as.

You were coming up, that was the thing.

Yeah, if, like, yeah, as I was coming up, it was like, okay, if you're, for example, if you're like a dev rel person, you're really good at marketing and your developer relationship. Yeah, yeah, like you're not. You're good at marketing. You could spike on marketing and you're a pretty good developer. You know, a lot of that, like people are, they kind of often they're.

Like elitism around it.

So like second class, like the pay scales are lower. I see. And you know, a lot of those people move into just software engineering because they're like, well, kind of hit a ceiling here. That's really weird because you know, if you're doing your own company, that is super. Yeah, so, so like, you know that yes, you should know how to build stuff and all that. But like that thinking is super important. So you know, it's been trained out of us, I would say, or trained out of engineers in a sense to like, okay, all that soft skill. Like, that's just soft bullshit. Like, that's not true. And so like, you know, it's an interesting like experience for me at least. Yeah, like, and so like, if I were to build a product, which I would try in the future, I think that all this, all this experience is very useful because like, yeah, it was kind of trained, like I was ignoring it for a long time because I was convinced, like, oh, okay. This soft stuff. Soft stuff. Like, focus on soft stuff. Then like, you know, whatever. It's not good. But yeah, you have to do everything as your own business. Right.

So, like, tell me about it.

Yeah. So it is like, really valuable actually. And like, you know, if you have some. If you have like, deep knowledge because, like, accumulated all this knowledge over some time working, plus all this consulting, it's like, okay, well, that's actually valuable. Instead of doing these like, very expensive consulting engagements.

One on one, lower impact.

Yeah, it's like one on one. Like, you know, it's like, okay, well, how can. How can we have like a consulting for everybody? You know? But that's what education is. Yeah. So it's like, okay, that's how I was like approaching it. It's like, okay, you know, it's selling an information product.

Yeah.

Now I will say though, that there is. So that's one side of the coin is like, it is valuable. It's interesting. It is like a thing, a product you can sell. The other side of the coin is to have some nuances. Like. Yeah. There is a ceiling. Like, you know, information product can only scale so much. And like, you know, it's always good if you can also have a product.

Yeah.

That provides value for your user because like, okay, you can teach someone how to do something, but can you solve the problem more in depth?

Yeah.

In their workflow with a tool. Okay. Then that. That is economically valuable and that can scale. Yeah.

Like, so is that what this Delphi test is about to see? Like, do people.

It's not really about that. No. I wouldn't say the. The hypothesis. I don't know what we're gonna learn. It's just like, you just want to learn. I'm just trying different experiments.

It seems like that's like something you love is learning. Have you always loved learning?

Yeah, I think so. Yeah.

Ever since you were a kid?

Yeah. Yeah, I tried. So I, like, I pivoted several times. Like. Yeah.

What was like Kid Hamill? Like, all the way to, you know, GitHub? Because we covered GitHub and beyond.

Yeah. So give me broad strokes. Okay. Yeah. I didn't really, like, do too much exploration until, you know, so I graduated college. Like, I went to go work at a bank and build credit risk models. So it was machine learning. I wasn't like, trying to do machine learning. It was just like, I had like a math major and like, all this statistics stuff. And you know, what about even before.

That, what made you pick math and statistics?

I was told To.

Oh, yeah.

But it was, like, actually, like, parents. Yeah, Parents. But actually, like, they were right.

Wait, did you have immigrant parents or.

Yeah.

Oh, no.

But I actually liked it.

Yeah.

But I didn't, like. No. So that was, like, one weird thing is, like, I didn't explore.

Yeah.

Stuff they, like, said, well, you should do this. Seems like you like it. And then they did say, it seems like you like it.

Okay.

And I was like, okay.

Credit to them.

I was like, oh, yeah. It kind of maybe feels right. So, like, you know, I did it. Okay. And it felt like at home. It was like, where the nerds were and kind of whatever. It felt like, familiar.

Jived.

And then I was like, you know what? I never, like. I haven't tried anything before. Like, else. Like, I'm just, like, all this, like, technical stuff.

Yeah.

So then, like, eventually I went to law school. It's like, let me just be a lawyer. Let me just, like, forget all this and let me, like, go do law. Let me do law now. Then, like, I discovered, like, I hate it.

What made you hate it?

It's just everything about is, like, sucks. Like, you know, you have to. It's all about, like, tradition. Yeah. Precedents moving, you know, everything moves slow.

Oh, yeah.

It's very.

Yeah, you would hate that.

It's very credential based.

Yeah. Not, like, merit.

Yeah. It's not really merit.

I see.

They wouldn't say that. They'd be. Oh, yeah, where's the merit? Because, like, it's about what school you went to and what. But that's not merit. I don't know. No. So, like, so I hated everything about it. I was like, you know what? I was like, yeah. I kind of liked what I was that. All that technical stats.

Mom and dad.

So, like, yeah, I went back to it. Cool. Okay. It's like, very expensive lesson of, like, he, like, was very hard to, like, do that and then say, I hate it.

And then you dropped out and went back.

Yeah, I mean, I graduated and whatever. And then, like, you graduated law school. Yeah.

Then you were like, that went back to tech.

Yeah.

Okay. And that was expensive lesson.

Really.

Tell me about that area of your life real quick. Was that, like. Was that, like, hard? Like, just economically, emotionally?

It was really hard. Yeah.

Tell me about that.

I was like, oh, yeah, it's gonna be good. Like, I'm gonna, like, do this thing, you know? Yeah. It was like a good law school. University of Michigan. Ann Arbor is, like, good. Like, I was like, oh, I might get, like, a law firm job. I'LL have good money. I don't really know. Like, I was convinced that, like, I like doing anything. I don't care. Like, yeah. If it's like, you know, I can get interested in this work. Yeah. Because I didn't really explore too much when I was a kid. I was just whatever. And, yeah. It was the first time I learned, like, what it feels like to hate something. Wow. Yeah.

And do you still hate that thing more than other things?

Like, oh, yeah. I hate it more than anything.

More than anything.

So there's. It's actually, like, really useful because, like, anytime there's, like, I joke with, like, there's other people that I went to law school with that hate it.

Yeah.

Just as much. But some of them are doing law.

Yeah.

And we talks every so often and we're like, look, anything that happens in life, like, we just think about. Remember that time we were in law school? Like, how much we hated that and how much we hate the law. And we're like, everything else is just amazing.

All fucking gravy from here. It's good to have those under your belt, man. To like, be like, oh, it can't be worse than that.

So I was like, you know, then I could, like, really invest in learning. So, like, then I invested a lot more in learning.

I see.

I was like, oh, no. Like, I do this.

Yeah.

Like, now I'm like, oh, this is, like, now if I hate something, I'm like, I really like this.

That's, like, way flipped.

Yeah.

Okay. Wait. That's amazing. I'm glad we got into.

So then I started, like. Then I was like, okay. Then I got deeper into it. Then I, like, got some more formal education. Yeah. In, you know, like, machine learning, Computer science. What was the formal education? Just like. Yeah. People that don't know. It was like a master's, you know.

Yeah.

In computer science.

Cool.

Okay. And so, like, I decided to go back. I just kind of wanted to wash my hands of the law. The law nonsense. And at that time in my career, I was like, oh, I just try to, like, pivot back and I needed to wash my hands. Yeah. Yeah. You know, like, degrees.

Is it different?

We're important back then. Or maybe at that stage or something. Yeah. Yeah. But, yeah, then it really, like, flipped when I got into Airbnb, cuz, Like, I guess, like, it takes sometimes one Silicon Valley company on your resume to flip things. Yeah. So, like, that was really. That was the flip. The moment I got a job, Airbnb, like, three months later, everyone.

Everybody's Hitting you up.

Yeah. Oh, wow. And then. Yeah.

And then GitHub came from that later.

Yeah. Then, like, I decided to go to GitHub, which is cool. Like, GitHub I did. So GitHub was like, you know, kind of an interesting, weird situation.

Why was GitHub weird?

Yeah, it's because, like, it was at a time where they didn't have a CEO and it was kind of in between things, like, drifting a little bit. It was. I liked my teammates a lot and, you know, the company was not. It's kind of not sure if it's a direction, but had, like, an amazing data set.

Yeah.

Had amazing culture. And so, you know, I didn't have a manager for, like, almost three years.

That's so fun.

And so what I did is, like, I did a lot of open source. Cool. So I worked really hard on open source stuff.

Yeah.

The whole time. I didn't just sit around. I actually worked, like, harder than I've ever worked before. I was like, okay, I don't have a manager. Okay. I'm gonna make an impact through working on open source ML stuff that is, like, also, like, somehow also touches upon GitHub things.

Yeah.

Like, in some way, I'll try to make it, like, related by one way or the other.

Totally.

And so, like, I met so many people doing that. That's when, like, I think I met so many people around. That's when I started blogging a lot. Okay. When I was doing that. Yeah. So I just say I just, like, wrote a lot of code. Wrote a lot of words. Prose.

Yeah. Yeah.

And that really helped. I don't know. I wasn't doing it for a reason. I was just like. Okay.

It sounded like you were just learning and wanting to make an impact.

Yeah.

And that compounded later, unbeknownst to you. It compounded later.

Yeah, Got it. That kind of set up for whatever.

For getting the first, like, services gig and.

Yeah. Helped with, like, everything, basically.

Yeah. So I want to go back to. Okay. So I want to go back to something you said, the chorus boys thing.

Yeah.

And you speak to something that is not spoken to a lot, which is Silicon Valley, like engineering tech, people like elitism. This is something I see too. So it's interesting to talk about. How would you define success?

That's a deep question. I could talk about what it means to me.

Exactly.

Potentially. I could try. It changes so often. So to me, it's basically about freedom. When I wake up in the morning, can't do. I feel like I have autonomy over My own schedule concretely, can I look if my calendar. Is my calendar empty? My calendar is empty. I feel very successful. Yeah. So how can I get to that? And so that's kind of the, that's the goal gradient kind of flows from there. Like how can you not have any calendar? What, how can you like how is that even possible? Yeah, okay. I don't know but like let's try to figure it out. Like you know and so certainly like having a job is not going to help with that. So like can you can do something else? I mean that's the immediate term. Yeah. Like that might not be the success metric like forever but it's just like at this point like that feels like an amazing goal. Yeah. Because I absolutely hate like seeing my calendar stack back to back with just like meetings.

Yeah, yeah, yeah, yeah, yeah, you've told me this. Yeah, you told me this. Well, okay, so let's intersect that with other people's. We talked about social status. That's like a vector of success to some people. Like you spoke about how VC backed venture back thing. That's like top level social status, level perception of success and then all the way down course voice. So like does that matter to you?

No, because I don't think it, I don't think it does. Like I'm just learning and doing whatever the hell. Like and I'm okay with like labels. I'm thinking like if I do something I should do it to the best of my ability. Yeah. So like if I selling a course I should volume I should you know, have it be the best course I can possibly be and try to learn as much as I can from it. And like tomorrow if I do something else, do that, I don't really care. I think it's a little bit like misguided. Like I understand where it comes from. I understand a lot of grift out there, a lot of whatever. I think, you know, it's always. There's a lot of definitely a lot of brainwashing going on in the sense like if you engineers will make fun of engineers for being good at anything other than coding. So I have good friends. That may be a strong statement. Let me rephrase that. So I think engineers will be make fun of other engineers for spiking in talent on certain categories. So I have a good friend for example that's working at Meta and he's a pretty strong engineer, but he's also really good like really, really good at writing, really, really good at building communities, really good at some other things. And like he has gotten flack from his co workers on, you know, some people have called him the marketing guy. Right. Or like, hey, you're really good. You're like a marketing guy. That's not. And it pisses him off and it pisses me off. So I'm like, this is a multi dimensional person.

Yeah.

And he's. I'm not, you know, I'm sincerely meaning. He is a good engineer. He really is. Like, he's a pretty good engineer. You know, he writes lots of code. Like all ships. Yeah, yeah. And so, you know, engineers make fun of each other and they will defend. And like so, you know, at companies also, you know, they're Leetcode. Leetcode interviews.

Yeah.

Where you have to like study these, memorize kind of these like 200 things. Yeah. And you know, people accept that as a rite of passage. In a continued rite of passage even sometimes to transfer positions within a company you have to do another leak code, sort of. Whatever.

Yeah.

And you know, engineers will defend that to the teeth. They'll say like, well, you know, that's, you know, it's, it's useful because of XYZ and ABC and this is what we have or whatever. So you know, it's like, it's almost like engineers will defend the Matrix. You know what I mean?

Yeah, yeah, I get that.

Okay. They will defend the Matrix to the death. Like, what is it? Agent. The Smith.

Agent Smith.

Yeah.

Yeah.

So it's like, you know, you kind of have to decide like you have to see that.

Yeah, yeah.

If you want to get outside the Matrix.

Totally.

To say like, look, like the only path is not just to grind your way up this one path of like, you know, keep getting better at just engineering only.

Yeah.

And like then, you know, just get better engineering.

Yeah.

That, you know, that is a, somewhat of a narrative.

Yeah, I see.

Somewhat of a kind of a pull, let's say. Yeah.

I could see that as like a Matrix thing that reinforces itself through social structures.

Yeah, it does reinforce itself. Yeah. And I would say like, you know, it can be hard to break free of that. It's also, you know, I would also argue like having a paycheck is the most intoxicating potent drug that one can take. Yeah. It's very difficult for people to get off of it. Yeah, it's incredibly difficult.

The hedonic.

You have to rewire your mind.

Yeah.

And you kind of have to like pry people off. And sometimes people can't get off of it unless they're forced off by some and they'll often tell you like, this is the best thing that ever happened. I was laid off. Yeah, I was XYZ some stories like that, whatever. And like this is what happened. And so, you know, so that's kind of like you have to keep that in mind when talking about these like labels. I think these, sometimes these labels like keep people oppressed.

Absolutely true. Absolutely true. In many other domains of life as well.

Probably so. Probably not just this. Probably just not this. But what I'm saying is like, okay, you kind of have to do things with like a good intention and not worry too much about some kind of social status. Because I think it just like is destructive. Yeah.

I think, you know, I appreciate you sharing your definition of success and like sharing more of this narrative which I think is like slightly toxic, you know, or it can be.

It's very toxic.

Yeah, it can be to like the person where it's like they want to do different things but then they feel this like self reinforcing pressure from society. But what would you say to that person that does feel that way, that feels like, oh man, I'm good at engineering, but I'm also good at these things and I'm getting this pressure from my peers. What would you say to them?

I would say like, yeah, just invest in yourself. And if you are good at those things, those things are super high leverage. If you look at a lot of founders out there, they are good at those things.

Yeah.

Or they should be. There are. They do get good at them or.

They have to, otherwise you won't be your business.

And so like it might not suit the narrative being in the like especially employee. It may not suit being employee. Yeah, right. Yeah. Because like if you are an employee and like you're good at these things, whatever. Yeah. It might, you might not fit inside this box. Yeah, right. Yeah. But that's just the system, you know. Yeah. This is not going to. Yeah. Like it makes sense. Like if you're good at all these things, like marketing, engineering, all this stuff, like maybe shouldn't have a job.

Maybe you don't fit in a J.D. right. In the job description. Yeah, I see. Okay, that's really helpful.

And understand.

So now that we talked about success, what do you think of failure?

Yeah, I think failure is really a good way to learn, you know, and so I think it's necessary, like almost. It's hard to do anything without failure. Yeah. You know, so like I have my own business, right.

Yeah.

And I have so many failures, like you know, hiring people, I like Waste so much money doing the wrong marketing tactics or hiring the wrong agency.

Going to law school.

Yeah. No, I mean, yeah. And those are different. Like, yeah. So like, and every time I hire the wrong agency or partner with the wrong person at times, or hire somebody, do the wrong thing or, you know, buy tools that even. Buying tools that don't make sense, I'm like, okay, I learned something. Like, yeah, okay, I got smarter now. Like, I, you know, I'm gonna be more savvy when, like, doing this XYZ thing.

Yeah.

You know, so I'm currently exploring, like, you know, how do I take, like, do better video editing? But I don't want to do the editing.

Yeah.

I want someone else.

Oh, we were just talking about this.

Yeah.

It goes deep.

So it's like, okay, like, how do I hire the right people for that? Yeah, yeah, yeah. So this is like a safe way of saying, like, that's like, very clear. Like, your failure is actually helping you. If you're. Because you're learning directly from them as learning.

I always say that failure only exists when effort is non existent. When you don't put in the effort to recognize what you got out of it, then, yeah, you could just wallow in your own shit and, like, think you fail. I've certainly had times like that, like, sure. Where it's like, oh, this sucks. I failed so hard. But then.

Yeah. I mean, it's even the law school thing to go back on that.

Like, yeah, tell me more.

So I was like, so when I was before law school, you know, I was working at this consulting firm called Accenture. And this happens at every consulting firm. And I, you know, and I worked at like, two different consulting firms. And they're both the same in the sense that they have their clients after they go work on their clients. So, like, you'll have dinner and you'll go laugh at your clients like, aha. Like, this person, like, doesn't know what they're doing. Yeah. Like, can you believe they're doing that? Blah, blah. And you get this. You get into. You're in a different matrix. Yeah. You're like, I can't. I can only do consulting. I would never go work at these companies or whatever. Right.

Yeah, yeah, yeah, yeah.

It's only until, like, I had to, you know, for whatever reason, you know, go to law school, whatever. To like, reset my brain.

Yeah.

Completely. But like, you know, those. There's people that are stuck in.

Oh, in the consulting sauce.

Yeah. Actually, when I went like, I checked on LinkedIn and like a really high percentage of people, like, just stayed in it, dude.

Yeah. I know some folks that.

And I don't. I. And I don't think it is, like, those people are convinced that was, like, the best thing financially. And, like, amongst, like, all kinds of other, like, points, I'm like, that's not true.

I. I think it's true for people that are actually happy, but when you talk to people and you get a sense of, are you actually happy? And then the ones that aren't are, like, you're convincing yourself you're happy. You're not happy. This is not. This is just consulting sauce. Yeah, they're lost in the sauce. Okay, that's so helpful.