Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)

One question that get asked a lot and a lot is how do we keep up to date with the latest AI news? Why? Why do you need to keep up to date with the latest AI news? If you talk to the users and understand what they want or they don't want, look into the feedback then you can actually improve the application way, way, way more.

A lot of companies are building AI products. A lot of companies are not having a good time building AI products.

We are in an ideal crisis now we have all this really cool tool. You have to do everything from scratch. You can have your design, it can have your write code, you can have your website. So in theory we should see a lot more but at the same time people are like somehow stuck. They don't know what to build.

All this AI hype, the data is actually showing, most companies try, it doesn't do a lot, they stop. What do you think is the gap here?

It's really hard to measure productivity. So I do ask people to ask their managers, would you rather have give everyone on the team very expensive coding agent subscriptions or you get an extra headcount? Almost everyone. The managers could say headcount but if you ask VP level or someone who manage a lot of teams, they could say one AI assistant. Because as managers you are still growing. So for you having one hr head scalp is big. Whereas for executive maybe you have more business metrics that you care about so you actually think about what actually drive productivity metrics for you.

Today my guest is Chip Huan. Unlike a lot of people who share insights into building great AI products and where things are heading, Chip has built multiple successful AI products, platforms, tools. Chip was a core developer on Nvidia's Nemo platform. An AI researcher at Netflix, she taught machine learning at Stanford. She's also a two time founder and the author of two of the most popular books in the world of AI, including her most recent book called AI Engineering, which has been the most read book on the O'Reilly platform since its launch. She's also gotten to work with a lot of enterprises on their AI strategies. And so she gets to see what's actually happening on the ground inside a lot of different companies. In our conversation, Chip explains a lot of the basics, like what exactly does pre training and post training look like? What is rag? What is reinforcement learning? What is RLHF? We also get into everything she's learned about how to build great AI products, including what people think it takes and what it actually takes. We talk about the most common pitfalls that companies run into where she's seeing the most productivity gains and so much more. This episode is quite technical, more technical than most conversations I've had, and is meant for anyone looking for a more in depth conversation about AI. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. And if you become an annual subscriber of my newsletter, you get a year free of 16 incredible products including Devin, Lovable, Replid, Bolt, N8M, Linear, Superhuman, Descript, Whisper Flow, Gamma, Perplexity, Warp, Granola, Magic patterns, recast, chat, PRD and mobin. Head on over to Lenny'snewsletter.com and click product Pass. With that, I bring you Chip when after a short word from our sponsors, this episode is brought to you by DScout Design Teams today are expected to move fast, but also to get it right. That's where dscout comes in. Dscout is the all in one research platform built for modern product and design teams. Whether you're running usability tests, interviews, surveys or in the wild field work, Descot makes it easy to connect with real users and get real insights fast. You can even test your FIGMA prototypes directly inside the platform. No juggling tools, no chasing ghost participants, and with the industry's most trusted panel plus AI powered analysis, your team gets clarity and confidence to build better without slowing down. So if you're ready to streamline your research, speed up decisions and design with impact, head to dscout.com to learn more. That's D S C O u T the answers you need to move confidently did you know that I have a whole team that helps me with my podcast and with my newsletter? I want everyone on that team to be super happy and thrive in their roles. Justworks knows that your employees are more than just your employees. They're your people. My team is spread out across Colorado, Australia, Nepal, West Africa and San Francisco. My life would be so incredibly complicated to hire people internationally, to pay people on time and in their local currencies, and to answer their HR questions 24. 7 But with Justworks, it's super easy. Whether you're setting up your own automated payroll, offering premium benefits, or hiring internationally, Justworks offers simple software and 24.7human support from small business experts for you and your people. They do your human resources right so that you can do right by your people. Justworks for your people. Chip, thank you so much for being here and welcome to the podcast.

Hi Lenny, I've been a big fan of the podcast for A while. So I'm really excited to be here. Thank you for having me.

I want to start with this table chart that you shared on LinkedIn a while ago that went super viral. And I think it went super viral because it hit a nerve with a lot of people. And let me just read this and we'll show this on YouTube for people that are watching. So it's this very simple table you shared of what people think will improve AI apps and what actually improves AI apps. What people think will improve AI apps. Staying up to date with latest AI news. Adopting the newest agentic framework. Agonizing what vector databases to use. Constantly evaluating what model is smarter, Fine tuning a model and then you have what actually improves AI apps. Talking to users, building more reliable platforms, preparing better data, optimizing end to end workflows, writing better prompts. Why do you think this hit such a nerve with people? And just what if you have to boil it down, what do you think is, what do you think people are missing about building successful AI apps?

One question that get asked a lot and a lot is that how do we keep up to date with the latest AI news? And I'm like, why? Why do you need to keep up your dick with the latest AI news? I know it's not very counterintuitive, but I just. So much news out there. A lot of people also ask me questions like how do I choose between two different technologies? Like maybe like recently like MCP versus like agent agents, right? Like protocol. And it was like which one is better? Or like this or that. And, and I think a serious question you should ask them is like first like if how much of the improvement could you get like from like optimal solutions versus non optimal solutions, right? And sometimes they were like actually it's not much, right? And I was like, okay, if it's not much improvement, why do you want to spend so much time debating something that doesn't make much difference to your performance? And another question they ask is like, if you adopt a new technology, like how hard it would be to switch that out to another. And sometimes it were like, oh, I think it could be like a lot of work switching it out. And I was just like, let's say here's a new technology, it hasn't been tested by a lot of people and if you adopt it, you would be stuck with it forever. Like do you actually want to adopt it? Right? And maybe you want to think twice about, about like over commit to like new technology that hasn't been better tested.

I love your just Broader advice is just simple like talk to to build successful AI apps, talk to users, build better data, write better prompts, optimize the user experience versus just like what is the latest and greatest? What's the best model to use right now? What's happening in AI? Let me follow this thread of this idea of fine tuning and basically post training. There's all these terms that people hear in AI and I think this is going to be a really good opportunity for people to learn what we're actually talking about since you actually do these things, you build these things, you work with companies doing these things. And there's a few terms I want to sprinkle in through the conversation. But let's start with this one. What's the simplest way for someone to understand what is the difference between pre training and post training and then just how fine tuning fits into that. Just what fine tuning actually is.

Disclaimer. I don't have full visibility into what this big secretive frontier labs are doing, but right from what I heard, I think it's one is supervised fine tuning when you have demonstration data and you have a bunch of experts. Okay, here's a prop, right? And here is what the answer should be like and you just train it to simulate, emulate what the human expert would be like. And that's also what a lot of people would open source models are doing is they do it by distillation. So instead of having human experts should write really good selling upgrade answers to prompts, they get very popular famous good models to like generate the response to it and like getting this trained smaller model to emulate. So so sometimes you see people. So that's because like some I really appreciate open source community by the way. But like going from like have been able to train a model that can emulate a existing good model. It's very different from like being trained a good model, like an output for existing good models. So it's a big step there. So yeah, so we have my supervised fine tuning and another thing that's like very big. I'm not sure you have guests talking about it already but like reinforcement learning is like everywhere.

Let's pause on that because I definitely want to spend time on that and that's such a cool topic that's merging more and more in my conversations. But just to even summarize the things you just shared, which I think is really, really important stuff. So the idea here is a model essentially this algorithm piece of code that someone writes and say the frontier models are feeding it just like the entire Internet of content. And basically it's trying to test itself on predicting in all of the. Across all that data the next word. Essentially token is a simpler way, is the correct way of thinking about it. But a simpler way to think about it is like the next word in text. And as it gets it wrong, it adjusts these things called weights, essentially just like. Is that a simple way to think about it? Even though that's, even. That's just like very surface level.

So I think of language modeling as a way of encoding statistics, statistical information about language. Right? So. So let's say that you. We. We both speak English. So we kind of get a sense of like what is more statically likely. Like if I say my favorite color is, then you would say, okay, that should be another color. Like the word blue would be much more likely to appear than the word like end of table. Right. Because statistically blue is more likely to come up to. My favorite color is. So. So it's a sign. Get this. It's. It's a way of encoding statistical information. So like when language modeling, when you train a large amount of data, like you see a lot of languages, a lot of domains, so it can tell like, okay, UVA says this standard. Then if user do the prompts, then it could come like with the next most likely token. So by the way, it's not a new idea, actually a video. So the idea comes very, very old, like from the 1951 papers, like English Entropy. I think it's like called Shannon. It's a great paper and I think it reads a story I really like. It's from. Did you read Sherlock Holme the way?

Yeah, I read a few Sherlock Holmes books. Yeah.

Yeah. So. So this is a story of like when Sherlock Holmes was using this statical information to like have shown a case. So he was getting. So this, in this story there is somebody left a message with a lot of like stick figures. So Sherlock Holmes was like, okay, he knows that in English the most common letter is E. Then the most common stick figure must be E. Right? And then he goes, he starts like that he was, you know, he saw the, the code. So I think that language. So in a way it's like simple language modeling, right? But instead of like at a word level, he does this as like, like character level and token is something in between. Right? A token is not quite a word, but it's bigger than a character. So let's say we, we say token because it helps us like read what. Help us reduce vocabulary. Because with character is like Smallest amount of vocabulary. Right. So M5 has like 26 character, but words can have like millions and millions. Right. Whereas tokens, you can be able to like get like the sweet spot between the two. So let's say that we have the new word, like how to say like podcasting. Right? Let's say it's a new word, but it can divide podcast and ink so people understand. Okay, Podcast, we know the meaning. We know that ink is like, like a verb, like gerund, whatever it is. So we even know the word like podcasting. So that's why the token comes in. But yeah, that's like the pre tuning is basically like encoding statistical information of language to have you predict what is most likely. I think that most likely is the most simple way of doing it because it's more like building a distribution of like. Okay, so next token could be more like 90% of the time. It could be like a color. Like 10% of the time could be something else. Right. So you basically distribution. So language would like pick, like depending on your sampling strategy, like do you want it to always pick the most likely token or you wanted to pick something more creative, you know, so, so, so I think my sampling strategy, I think is something extremely important, so can have you boost up performance in a, in a huge way and very, very underrated.

Okay, awesome. So essentially a model is just code with this whole set of weights. Essentially the statistical model that has learned to predict what comes next after certain words and phrases.

Yeah.

And then post training and fine tuning specifically is doing that same thing. So pre training you get like GPT5. Fine tuning is someone taking GPT5 and doing the same sort of thing, adjusting these weights a little bit for specific use cases on data that they find find as necessary to do their very specific use case. Is that a simple way to think about it?

Yeah, I think of like width is like functions, right? So let's say just like you have maybe has a functions of like maybe Lenny's height is maybe like 1x, like 1x plus something like 2x, like 1 and plus something is a weight. Right. So you change it until you fit the correct data, which is like my height and your height. Right. So second thing is a width is just like a weight, like they function. So you like chain adjust the width so they can fit fit the data, which is the training data.

Awesome. Okay, so we're talking about pre training, post training, fine tuning. Is there anything else here that's important to share about? Just like, what? This is exactly what people need to Understand about these parts of training.

So the vast majority of time we don't touch on like pre training model. Like as users we don't use.

That's already done for us.

Yeah. So I think that actually is pretty fun like process. Like when my friends training models I try to play with their pre shooting model and they're horrendous. They're like saying things. It's like oh my gosh. It's like like yeah, it's crazy. So. So it is really interesting to look at like how much of like post training can change the motor behavior. Yeah. And I think that's where like a lot of time is that a lot of people are spending energy on nowadays in Frontier Lab is on like post training. Because pre training. I think so pre training have been used to like increase the general capacity of a model, capabilities of a model. And it depends on. You need a lot of data and model size to increase the model capabilities. And at some point we are actually have quite maxed out on Internet data and people text data very max out. I think a lot of people are doing with other data like audios and videos and everyone's trying to think of what is the new source of data. But we're like post training. But method of course is more of like everyone can have very similar pre training data is that post trading is where they make a big difference nowadays.

This is a good segue to. You talked about supervised learning versus unsupervised learning. I love we're getting into this. By the way, this is super interesting. So you talk about labeled data. Basically supervised learning is AI learning on data that somebody has already labeled and told it here is correct versus incorrect. For example, this is spam versus not spam. This is a good short story. This is not a good short story. We've had the CEOs of a lot of these companies that do this for labs. Merck Core and Scale Handshake. There's micro, there's a few others. So is that essentially what these companies are doing for labs, giving them labeled data, high quality data to train on?

It is in a way but I think it's more like a product of big equations. So there are a lot more different components than that. So that's why I was talking about reinforcement learning. I'm not sure if your CEO that you interview bring up that term. So the idea is that you want people to like. So let's say you have a model. Give the model like a prop. Right. And it produce an output. Right. You want to buy like you want to reinforce or encourage the model to produce an output that is better. Right? So now it comes to, like, how do we know that the answer is good or bad? Right? So each and everybody realize on signals. So one way to get like a response good or bad is like human feedback, right? They happen. We have two responses. You can, okay, this one's better than the other. And we do that is because, like, as humans, we tend to. It's very hard to give like concrete score, but it's easier to do comparisons. Right? Like if you ask me, okay, give this song a score. I'm not a musician, like, and don't know, like, how hard it is. Like, it's like, yeah, I don't know, like what, like 10. I go to six, you know, and then if you ask me again a month from now on, I completely forgot and say, okay, maybe now seven or like four. I don't know. But then if you ask me, okay, here are two songs. And which one could you prefer to play for the birthday party? I was like, okay, I can play it before this song. So, like, comparison is a lot easier. So say, yeah, so you have a human. You have human feedback. And then you use this human feedback to train a reward model. So you like, tell like which? Like, so. And then the reward model help you. Like, okay, it's a model that produces response. Is a reward model can score is this good or bad? Initializing bias toward like producing better model the better responses. Another way is that you can. Instead of using a human, you can use like AI, right? Like look as a response. Yes. So good, good or bad. Right. Orange, I think is that people are very big on nowadays like verifiable rewards, which is like natural. So basically they give it a math problem and then math solutions. Like, is a model output a solution? You know that. Okay, it's a expected response should be in a 4H2 and it doesn't provide 4A2. Then it's. Then it's wrong, right? It's not a good response. So, yes, like, a lot of time people like using this human labor like human. Human laborers should produce how to say expert questions and expected answers and in the ways that design systems that verifiable so that the models can be trained on.

Okay, I'm really glad you went there. This is essentially RLHF reinforcement learning with human feedback. Which is exactly what I wanted to also talk about. Right?

Yeah. So I think it's general. It's a way of learning. It's like training is conversational. Learning and whether it learn from human feedback or AI feedback or verifiable rewards, I think they say you say it's just different way of clipping signals.

Awesome. Yeah. We had the CEO of Anthropic on the podcast and he talked about their version of RLHF which is AI driven reinforcement learning. I love the way you phrased it where you basically you want to help the model, you want to reinforce correct behavior and correct answers. And this is the method to do it. Whether it's say an engineer seeing an output from a model, being like, no, here's how I would code it differently. And then training and it's training a different model that the original model works with to tell it am I correct or not correct. Is that right?

Yeah, I think that's a way of looking into it and I think that's a space is so exciting nowadays because there's so many like domain expert tasks that the model, like that model developers want model to do well on. Right. Let's say you're like accountant, right? Like maybe you want to use a model to have accounting tasks. So I need a lot of like accounting data, like examples for my accounting accountants. So you need to hire a lot of them to like do it. Or if you want a physics problem, I want to do, I don't know, like legal questions and stuff or like engineering questions or like somebody was telling me they want to do like using like coding for to solve scientific problems and not just like coding to build product which is another different whole realm of things. And I also like using very specific toolings like yeah, like I'm not sure what apps you use, but maybe like forating Apple or like QuickBooks or like Google Excel. Like they are very specific like tool specific expert expertise that you want the model to learn. So they need a lot of like humans experts in this area to like create data to train them. And it's a massive thing. It's like people because everyone wants a lot of data and like wants labs at like unlimited budget. But whether I think this is also like a little bit of low key interesting economics. So I'm not sure you've talked to like the guests about. I thought it's very interesting to think about because it's very lopsided, right? Because like they only like a very small numbers of frontier labs, right? And they want a lot of data and there's like a massive amount of like startups or companies that providing data. So like you can see this company is like this startup like doing data labeling that they have like maybe have like massive ar, but you ask them like, okay, so how many customers you have? And there could be like a very small numbers. I'm not sure. I'm not sure. I saw you smiling.

Yeah, yeah, we chat about that.

Yeah. So I'm like a bit like, look me uneasy right here. I have a company that's growing like crazy, but it's like heavily dependent on two or three companies. And at the same time, if I was this company, Frontier Labs, what could be the right economical things for me to do? Right now I want a lot of startups, I want to have a lot of providers so I can pick and choose. And as these providers can also like compete each other to lower the price. And it's so dependent on me, the good stands for me regardless. So I feel like. Yeah, so this economics, the whole economics is very interesting to me and I'm curious to see how it plays out.

What I'm hearing is you're bearish on the future of these data labeling companies because as you said, they don't have a lot of leverage over pricing because they have so few customers and there's so many people getting into the space. So basically, even though they're some of the fastest growing companies in the world, you're feeling like there's a challenge up ahead.

I'm not sure if I'm bearish on it. I think, I'm curious because I think things have, has a way of work out in ways that I don't expect. So I think that maybe these companies, they have a lot of data, maybe they wouldn't be able to use that to have some insight that helps them stay ahead of the curve, you know? So I don't know a very fair answer.

Okay, while we're on this topic, I want to chat about evals, which is a very recurring topic in this podcast. This is the other piece of data content these companies share that AI labs really need. Can you just talk about what an eval is, the simplest way to understand it, and then how this helps models get smarter.

So I think people approach eval, I think they're like two very different problems. One is a app builder, right. And like, can I say have an app that do, like maybe a chatbot? Very simple. I know it's the first thing that came to my mind and I want you to know is chatbot is good or bad, right? So they need to come up with like a valid chatbot. Another thing is I think of this as a task specific event design. So let's say, I'm a model developer, and I want to make my model better at curve writing, right? And it was like, okay, but how. How. How do I even measure code writing? Right? So I even need someone to, like, okay, understand curve writing and think about, like, what makes good story. Like, what makes a story good. And then designed the whole data set and then criteria to evaluate crib writing. So, yeah, so. So I think there's that. I think it's like more like eval design. That is very interesting. Coming with criteria, come up with guideline how to do it, and then also, like, train people, like, how to do it effectively. So I guess in Aquas, I think EVAR is really, really fun because it's extremely creative. I was looking at, like, different Evans people built and was like, wow, like, it's just not dry at all. This is like, super, super, super fun.

We had a whole podcast on evals with Hamill, Hamil and Shreya, and. And that's exactly what they talked about. It's just. It's actually really fun to create evals for. For companies especially. So let's still dig into that one a little bit more. There's this kind of debate online that I don't know how big of a deal this debate is, but it feels like people spend a lot of time thinking about this. This idea of, do we need evals for AI products? Some of the best companies say they don't really do evals. They just go on vibes. They're just like, is this working well? Can I feel it or not? What's your take on just the importance of building evals and the skill of evals for app. AI apps, not the model companies.

You don't have to be like, absolutely perfect at things to win. You just need to be good enough and being consistent about it. Okay. This is not a philosophy I follow, but I have worked with enough companies to see that play out. So when I say why company don't evaluate? Let's say you are an executive and you want to have a new use case. So here's a use case. You started out, we built, and it's like, it works well. The customers are somewhat happy. You don't have the exact metric for it, but so traffic keeps increasing. Like, people seem happy, people keep buying stuff, right? And now here's our engineer coming, like, okay, we need EVA for it. And so it was like, okay, how much effort do we need to put into eva? And they were like, okay, maybe like two engineers as much as much. And they could maybe would improve and was like, okay, so how much expected gain can I get from it? And the engineer would be like, oh, maybe you can improve it from like 80% to like 82%, 85%. Right? And I was like, okay, but who are you taking is that two engineers and be able to launch a new feature. Then it could give me so much more improvement. Right? So I think it's like one of them is avar. Sometimes people think of evar. It's like, okay, is this good enough? Just touch it. If you do spend a lot of energy on evar, it would only incremental improvement where it spends the energy on another use case and maybe get it good enough that you can vibe check it. So I do think just like, maybe that's a debate is about. I do think just like a lot of time people just like get things to the place when it's like, okay, good enough, people run. But. And then, but of course it's like there's a lot of risk associated with it because if you don't have a clear metric, you have good visibility to how the application tomorrow is performing. It might do something very dumb or it can cost you. Like I know some. Something like crazy can happen. So. Yeah, so, so, so I do think eval is very, very important. If you have, if you operate a scale and where like failures can have like catastrophic consequences, then you do need to be very tyrannical about like what you put in front of the users. Understand different failure modes, like what could go wrong. And also maybe in a space when that like it's is a feature the product is as a competitive advantage. Right. You want to be the best at it. So you want to have like a very strong understanding of where you are and where you are with the competitors. But it's just something that's more like a low key. Okay. Just like something it's like, okay, that's not the core, but it helps with our users. Then maybe you don't need to be so obsessed or Tyrico about it and say, okay, that's good enough for now. And if it fails, then it fails. Okay. I know it's like, it's so terrifying. But yeah, I think it's all about the question of return on investment. I'm a big fan of eval. I love running eval and I say it's like I understand why some people would choose to not focus on eval right away and choose bringing on new functionalities instead.

Awesome. That is a really pragmatic answer. What I'm Hearing is evals are great, very important, especially if you're operating at scale. But pick your battles. You don't need to write evals for every little feature. Something that Hamill and Shreya shared is that people need just like, I don't know, five or seven evals for the most important elements of their product. Is that what you see or do you see a lot more in production that people build and need?

I don't think of like just a fixed number on like the evals. Like, what was the goal to eval? Right. The goal of evolve is to guide the product development. So, like you see evolve, because I think I'm a big fan of evolve, is that it helps you uncover opportunities where the product is doing well. So I sometimes I've seen it very obvious. Okay, so we say, okay, we look at the API, we realize it's like, okay, it performed really poorly on this, like, specific segment of users. And then we look into it. It's like, okay, what, what, what, what, what's wrong with it? And it turns out it's like we just like, don't have a good messaging to it. So like, we should like, just focus on the things that we do. Polio can improve significantly. Yeah. So I kind of like, the number of eval is really depends. Like, we have seen product with like hundreds of different metrics. People are going crazy. This is because that product is like general, right? You have different type, like on evar for like, I don't know, like verbosity. Have like one EVAR for like user sensitive data and like another is like, for length. But like has a number of like, okay, let's just pick a good example, concrete example, like deep research. So you have the application, you have like builds a model to like do deep research for you. Right? Like, okay, like have a prompt. Let me say, okay, do me a comprehensive research on online podcast and help me like some like propose, like show me report on what kind of topics he's interested in, what kind of videos could get the most views, or like what topics that he's missing on that he should be covering. Right. Like have this kind of like prompt. Then how do you evaluate the result? Right? I don't think this like one metric that would help. Maybe it's just like, maybe you have like 100. I think somebody has a benchmark and is to get like 100 expert, like write a bunch of prompts and I go through like all the, all the answers on AI and I do it and it's like, it's extremely Costly and slow. Right. But if you might have something else for some more, like one way, I was thinking about it, I was talking to a friend about it, and one way is like, how do you produce the result of the summary? Right? At first you need to do, like, gather information. And to gather information, you need to do a lot of search queries. You gather graph the search results, and then some of the search results you aggregate. And then maybe say, okay, I'm still missing on this. You have to do another route and another route and is there any other summary? So every step of the way, you need evaluations. You don't need to the end to end. So maybe for the search query, you may first think about like, okay, now I write five search queries. Am I looking to like, how good are these search queries? Like, do they like, are they like, similar to each other? Because you need five search queries that are very similar. Like, okay, learning podcast. Then many podcast last month lend a podcast like two months ago, right? It's not. It's not very, very exciting. But like, if the quality is a podcast, like the. The keywords are like more. More diverse, right? And then look at the results of the, of the search query. And they say you enter the search query, like Lenny postcard data labeling. And then they come up with like 10 pages, 10 results. And then you come up with like, oh, Lenny podcast on. I don't know, I don't know, like Frontier Labs. And you have like 10 results. And my look was a different web page. Like, how much of them overlapping? Like, I. We are we doing both, like the breadth, like getting a lot of page, but also like, do we have depth and also like, have relevant. Because we come up with the search queries that are completely irrelevant to the original prompt. So I feel like every aspect of it, it would need a way of evaluating. Right. So I don't think it's just like, how many evolve should I get? But like, how many evals should do. I need to get a good coverage, a high confidence in my application's performance, and also to help me understand, like, where it is not performing well so that I can fix it.

Awesome. And I'm hearing also, just especially for the very core use case, like, the most common path people take in your product is where you want to focus.

Yeah. So yeah.

Okay, let me. There's one more term I want to cover and I want to go in a somewhat different direction. Rag. People see this term a lot. Rag. What does it mean?

So RAG stand for retrieval augmented generations and Also not specific to generative AI. So the idea is just like follow the questions, we need context to answer. So I think it came pretty, I think it's from the paper 2017. So someone was like. So they realized it's like for a bunch of benchmark, when the question answering benchmarks, they realize it's like, okay, if we give the model information about the questions, then the answer can be much, much better. So what they do is try to retrieve information from Wikipedia. So for questionable topics, just like retrieve that and then put into the context and answer it does much better. So I feel like it sounds like a brainer, right? I mean like, obviously. So I think that's what Racket as a simplest sense, it's just like providing the model with a relevant context so that it can answer the questions. And that's where like things get like really more interesting. Because traditionally when it started out, rack is mostly like text. So we talk about a lot of ways like how to prepare data so that the model can retrieve effectively. Let's say there's like not everything is a Wikipedia page, right? Wikipedia page is pretty contained and you know, okay, everything about it is about a topic. But a lot of times you have documents extremely long, right? And they have a weird way of structures and documents. Let's say that you have documents about Lenny podcast, right? And in the future, in the beginning of documents, like from now on podcast wouldn't refer to Lenny's podcast, right? So let's say somebody in the future is like, okay, tell me about landing, right? Many work. And because as a reference document does not have the term many, you just don't know. You might not read through it and the document is long enough that it chunk into a different part. So like the second part doesn't have the word vanity, so you cannot reach it. So you have to find a way to process data so that makes sure that it can retrieve the information that's relevant to the query, even though it might not immediately obvious that it's related. So it will come up with only thing of writing, like contextual visualization, like giving a chunk of the data the relevant, like maybe in a summary metadata so that it knows or like some people use it like as a hypothetical question. It's very interesting, like for even this chunk of like documents, I must generate a bunch of questions that the chunks can help answer so that when you have a query it's like, okay, does it match any of the like hypothetical questions so you can fetch it. So it's very interesting approach. Okay. So maybe before I go to the next thing, I just want to say that's like data preparations for RAC is extremely important. And I would say that's like in a lot of the companies that I have seen, that's like the biggest performance in their rack solutions coming from like better data preparations. Not agonizing over like what better database to use. Because it's very important to care about like things like latency or like, if you have like very specific access patterns like read heavy or write heavy, of course it's like it matters. But in terms of pure quality answers, right? I think the data preparation just like hands out.

When you say data preparation, what's an example to make that real and concrete for us to understand?

So one way is mentioned as in you have chunks of data so we can think about how big of each chunk should be, right? Because if it's sort of thinking about if the context you want to maximize, maybe you can. It's very simple. Example. Right now you want to retrieve like 1,000 words, right? So if each chunk of data is too long then, so if a data chunk is long, then it's more likely to contain more relevant metadata, so you can retrieve more. But if it's too long, like, then you have a thousand words. And so chunk is like a thousand words. You can reach one chunk. So it's not very useful. But if you choose short, then you can retrieve more relevant information. Like, oh, so it can retrieve a wider range of documents and chunks, but at the same time each chunk is too small to contain relevant information. So we have very nice chunk design, like how big each chunk should be. You add contextual information like summary metadata, hypothetical questions. Somebody was telling me just like a very big performance I got is that from rewriting their data in the question answering format. So like instead of having like. So they have a podcast, right? Instead of just chunking the podcast, you just like reframe, rewrite it into like here's the question, here's answers, like, and produce a lot of them. It can use AI for that as well. So that's one example of data processing. A lot of example we, I see is like for people helping, like using AI to have like specific tools and documentations, right? And a lot. And we write documentation, usually a lot of documents. Documentation today is written for human reading. And AI reading is different because it's different because humans, we have common sense and we kind of know what it is. So one things are like human. For human experts, they have the context that AI doesn't quite have. So somebody told me that the big change they have is let's say that you have a function, the documentation for this maybe the library. And the library says okay, the output of this one is like maybe talking for like I know some crazy term, maybe some temperature or something under grab it should be like 10 or minus 1. And as a human expert maybe understand the scale, like what one in the scale mean but like for AI just really doesn't understand what that means. So actually have like another annotation layer for AI. It's like okay, you've got temperatures equal one mint like that. It's not like it's absolute temperature, it's like absolutely associated with the scale over there. So it's just saving all this data processing to make it easier for AI to retrieve the relevant information. To answer the questions, this episode is.

Brought to you by Persona, the verified Identity platform helping organizations onboard users fight fraud and build trust. We talk a lot on this podcast about the amazing advances in AI, but this can be a double edged sword. For every wow moment, there are fraudsters using the same tech to wreak havoc, laundering money, taking over employee identities, and impersonating businesses. Persona helps combat these threats with automated user business and employee verification. Whether you're looking to catch candidate fraud, meet age restrictions, or keep your platform safe, Persona helps you verify users in a way that's tailored to your specific needs. Best of all, Persona makes it easy to know who you're dealing with without adding friction for good users. This is why leading platforms like Etsy, LinkedIn, Square and Lyft trust Persona to secure their platform. Persona is also offering my listeners 500 free services per month for one full year. Just head to with Persona.comLenny to get started. That's with Persona.comLenny thanks again to Persona for sponsoring this episode. Awesome. Okay, so you've talked a bit about how you work with companies on these sorts of things, on their AI strategies, on their AI products, how they build, which tools they build, all these things. I want to spend a little time here because a lot of companies are building AI products. A lot of companies are not having a good time building AI products. Let me ask a few questions along these lines of what you've learned working with companies that are doing this. Well, one is just, I guess in terms of AI tool adoption and adoption general within companies. There's all this talk recently of just like all this AI hype, the data is actually showing. Most companies try. It doesn't do a lot. They stop. And so there's all this just like maybe this isn't going anywhere. So in terms of just adoption of tools and AI within companies, what are.

You seeing there for gen AI in company? I think there are two types of gen AI toolings that have been. I've seen like one is to like internal productivity, right? Like have coding tools like chatbot internal knowledge. Like a lot of big enterprises have some kind like a wrapper around like models but like with access like maybe some different type of rack solution. I think we talk about okay, like text based rack. We haven't talked about agent rack or like having so like multimodal rack yet. But it's like yes, there's a whole very exciting area around that. Yeah. So like basically to allow the employees to like access internal document. Somebody, somebody asks like okay, I'm having a baby, what could be the maternal or partner policy, right? Or like am I having this operations? Could the health benefit like cover that? Or like I want you like interview. I want to like refer my friend. What could be the process for that? So a lot of this like having chatbot internal chatbot to help with internal operations and another things, another category is more like customer facing so or like partner facing so. So customer support chatbot is a big one. If a hotel chain you might have like a booking chatbot which is like somehow massive. Like a lot of booking chatbot because I guess it's, it's, it's. I do have this theory of like a lot of applications companies pursue because they can measure the concrete outcome and I feel like booking on a sales chatbot is very clear, right? Like what's a conversion rate right now with a chatbot with human operators and what could be conversion rate with a chatbot. And somehow I think it's like very clear outcomes and companies are easier to buy into these solutions. So a lot of companies have that customer facing chatbot. So yeah, so that is another category of tool and I think that for customers or external facing tools because people are driven to, people are driven to choose applications with clear outcomes. So the questions of adopting them is really based on whether they see the outcome or not. Of course it's not perfect because sometimes the outcome can be bad. Not because the idea or the application's idea is bad, it's just because the process of building it is not that great. So it's tricky for the internal adoptions of tooling internal productivity. That's where it gets tricky. I would say like a lot of companies, what they think of AI strategy. Like I think of AI usually have Very have like two key aspects, right? It's like use cases and the second is talent. You might have a great data for great use cases, but you don't have talents and you cannot do it. So a lot of time at the beginning with Gen AI and it's still. And sometimes I'm really admired a lot of companies for that. It's just like, it's like, okay, we need our employees to be very Gen AI aware, like very AI literate, right? So what they do is I start like maybe like adopting a bunch of tools for, for the team to use. They have a lot of upskill, upskilling workshops. Like they encourage learning. I think it's like a really, really good thing. And it's also willing to spend a lot of money into like adopting like giving people like ChatGPT subscriptions, personal subscriptions, cloud code subscriptions, like to get the employees to like to, to be more AI literate. And that's the thing is like a lot of the secretary come to me and say, okay, we spent a ton of money on this tooling. But then we don't see. Because you can see the usage and it's like, but people don't seem to use them as much. And what is the issue? So, so yeah, so I think that that is, that is tricky. Yeah.

What do you think is the issue? Is it just. They're not, they're like, they don't know how to use them. Like, what do you think is the gap here? Do you think we'll get to a place of just like, wow, work is completely different because of AI. For a lot of companies, the main.

Thing is like it's really hard to measure productivity again. So I talked to a lot of people on their website. First of all, on one example is coding, right? A lot of companies not using coding agents or coding, AI coding. And I was asking, I was like, I was like, do you think that like it helps with your productivity? And a lot of times the question is very hand wavy. It was like, okay. They were like, okay. I feel like it's been better, right? And I said okay because we have more PRs, we see more code and then immediate correctness. Okay. But of course, code number of live code is not a good metric for that. Right? So, so it's really, really tricky and it's something funny. So, so, so I do ask people to ask their managers because I work with like usually VP level, so they have like multiple teams under them. So I ask them like, okay, do you ask a manager like, okay, would you rather have access? Would you rather have give everyone on the team like very expensive coding agent subscriptions or you get an extra headcount? Right, let's say this like maybe like and almost everyone could say, the managers could say headcount. But if you ask VP level or like someone who manage a lot of teams, they could say just like they could want AI assistant assistance tools. And the reason is that people say like okay, because as managers, right, because you are still growing, you're not as a level when you manage hundreds of thousands of people. So for you like having one hr hash count is like, it's big. So you want that not for productivity reasons, but because you just want to have more people working for you. Whereas for executive you care more about, maybe you have more like business metrics that you care about. So you actually think about what actually drive, drive productivity metrics for you. So, so yeah, so it's tricky and I think just like the question of like productivity, it's not, I'm not sure it's like fundamentally there's some people more productive, but it's just like we don't have a good way of measuring productivity improvement. Another thing is also very widely and I think that people do tell me that they notice different buckets of employees, like different reactions to AI assistive tools. Like first of all, I keep going back to coding because it's big and it's easier to reason somehow. So it says I have different reports. One team would tell me that one of people tell me, okay, amongst all his engineers, he thinks it's like senior engineers would get the most output, would be more productive because it's like okay, so that person, very interesting. So he actually divided his team to like three buckets. But he didn't tell them obviously he was like, okay, here's more like currently like best performing, average performing and lowest performing. And then there's a randomized trial. So like they give like half of each of each group like access to like to like cursor. And then he was noticed like over time was like okay, the something funny like the group that get the biggest performance boost. Like in his opinion, Segu is very close. In his team, there's the biggest boom boost, like the senior engine, the highest performing. So the highest performing engineer get the biggest boost out of it. And then the second group is just like the average performing. So his opinion is okay, the highest performing engineers is a normal proactive. They will say no how to solve problems. So AI helps them solve problem better. Whereas the people who are Typically lowest performing, they only don't care much about work. Right. So like this easier to just like go on autopilot, get it to like generate like that code and just like do it. And I just don't know how to do it as another company, however, they tell me just like actually senior engineers are the one most resistant to like using AI as tooling. Because I said it's like okay, but AI, because they are more opinionated and they have very high standard, was like okay, but AI code, J code just sucks. So just like very, very resistant in using this. So I don't know, I haven't quite be able to reconcile very different reports on that yet.

This is so interesting. So just to make sure I'm hearing what you're the story. So there's a company work with that did a three bucket test with their engineering team where they created three sorts of groups. The highest performing engineers, mid performing engineers, lowest performing engineers. And gave some of them. So they gave some of them access to say cursor. Was it cursor or what did they give them access to? It was cursor.

I think by side was cursor.

Okay, cool. And so within.

I didn't work with them. This is more like a friend company.

Okay. It's a friend's company. So did they give like half of the higher performing engineers cursor and half not or how did they do the split?

Yeah, so like they give like half of the entire company but like half for each bucket. Yeah. And then they observe the difference in like productivity.

I see.

Yeah.

So how do they even do that? They're just like, okay, you get cursor, you don't get cursors. How did they do that? That's so interesting.

Yeah. I didn't get just the mechanics of it, but I was like at respect here for doing a randomized trial.

That is so cool.

Yeah.

Okay. Wow. How large was this engineering team? Was it like hundreds of people?

It's not that large. It's about like maybe 30. 20. 40.

Yeah, 30 to 40. Okay.

Yeah.

Wow. Okay. So they found that the highest performing engineers had the most benefit from using AI tools. And then behind them was the middle tier engineers and the worst performers were the lowest performers.

But it's not the same everywhere. Like, Right, Different.

Right. This other example you shared of just senior engineers in this one example are most resistant to changing the way they work, which I get. I do feel like the most valuable people right now other than ML researchers and AI researchers like yourself are senior engineers because it Feels like junior engineers are just like so much of this is now done by AI, but it's but an engineer that knows what they're doing, that understands how things work at a large scale with AI tools. Just basically like infinite junior engineers doing their bidding feels like an extremely valuable and powerful asset.

Yeah, I definitely really appreciate as you see companies like we appreciate engineers who have a good understanding of the whole systems and be able to have good problem solving skills like thinking holistically instead of like locally, locally. When our company have seen the way they work, as I said, told me it's like we're completely different now and like so they actually restructured engineering Org so that like they get more senior engineers should be more in the peer review because they like to get like sort of writing guidelines on like what is a good engineering practices. What is a process would be like maybe like okay, so they've write like a lot of like processes on how to work well and then they have more junior engineers who just produce code and submit pr but senior engineer more in the reviewing case. So I thought it might be prepared for the future. So another company actually told me something very similar. So it's like preparing for future when they only need a very small group of very, very strong engineers to create processes and reviewing code to get into production. But I get like AI or like junior engineers to produce code. But then the question becomes just like how does one become a very strong.

Right, that's right, that's right.

I feel like, yeah, so I don't know what's the process I was thinking about? Like yeah, no one's thinking about it.

It's just, it's a problem. We won't have any more in 10, 20 years there'll be no more engineers because no one's hiring engineers. Although I could make the case junior engineers, people just getting into computer science right now are just native AI. Native. And in theory, you could argue they will become really good really fast if they're curious, aren't just you know, delegating learning and thinking to AI, but learning how to actually using it to learn how to code well and architect correctly. Like you could argue they will be the most successful engineers in the future.

I do think that what I mentioned is I load into architect. I think I grouped that in like system thinking. I do think it's a very important skill because I think AI can help automate a lot of like distributed skills. But like knowing how to utilize these skills together to solve a problem, it's hard. So this is a Webinar between Mehran Sami was one of my favorite professors. He was a chair of the curriculum at the CS department at Stanford. So he spent a lot of time thinking about CS educations, right? Like what, what, what should students learn nowadays in the era of like AI coding? And then the other person is like Andrew Ng, which is of course is like a legend in the AI space. And ne Sami person like Sami said something very interesting. Like he said like a lot of people think of CS is about coding, but it's not like coding is just a means to an end. Like CS is about system thinking. Like using like coding to solve actual problem and problem solving problems will never go away because what AI can automate more stuff, the problem is just get bigger but the process of understanding what caused the issue and how to design step by step solution to it will always be there. So I think an example of I actually have a lot of issues with AI in the way of is debugging. So I'm sure you use a lot of AI for coding. But something I've noticed and also seen from my friends, it's like it is pretty good when you have very clear, well defined tasks. Maybe write documentations, fix specific features or build an app from scratch. Doesn't have to interact with a large existing code base, but you added something a little bit more complicated. Maybe require interaction with other components and stuff. It's usually not that good. For example, I was using AI to deploy applications and it was testing out a new hosting service I was not familiar with. It was okay. Usually they form me. So I think AI does give me is like confidence to try a new tool like before what AI is like trying new tools. Has written out documentations from the beginning but AI was like okay, just try it out and learn. So I was testing out this new hosting service and it kept getting a bug that was very, very annoying. And it was like okay, I asked car codes like fix it and it keep giving it keep changing the way like maybe change environment variable, fix the code, maybe I change from the function to this function, maybe change the language, maybe it doesn't process JavaScript, I don't know, whatever. And it didn't work. And it was like okay, that's it. I'm just going to read the documentation myself and see what's wrong. And it turns out it's like I'm on another tier. Like the feature I want did not is not available in this tier. Right. So I feel like, okay, so the issue with cloud code is just trying to focus on fixing things from a very a different component versus the issue is from a different component. So what to make I think of like, okay, be understanding how different components work together and where the source of issue might come from. You need to give a holistic view of it. And it's made me think it's like, okay, how do we ditch AI like system thinking like that. I think I have all the human experts having right like very much people call this scaffold just like, okay, for this kind of problem, look into this, look into that, look into that and then stuff. So I think that could be one way. But that's what made me think. It's like how do we teach humans my system thinking? Yeah. So I think it's very interesting skill. I do think it's very important.

That's exactly the same insight Brett Taylor shared on the podcast. He's the co founder Sierra, he created Google Maps, he was CEO of Salesforce, Quip a few other things and I asked him just like should people learn to code? And his point is exactly what you said, which is taking computer science classes is not about learning Java and Python. It's learning how systems work and how code operates and how software works broadly, not just here's like a function to do a thing. One thing that I wanted to help people understand. You wrote this book called AI Engineering, which is essentially helping people understand this new genre of engineer. And you have this really simple way of thinking about the difference between an ML engineer and an AI engineer, which has a really good corollary to product managers now of just like an AI product manager versus non AI product manager. The way you describe it and fill in what I'm missing is just ML engineers built models themselves. AI engineers use existing models to build products. Anything you want to add there.

One thing I really dislike about writing books is that you have to define like this. And I think it's like no definitions will be perfect because they're always really edge cases. But yeah, in general I think it's ideas like Gen I AI as a service, like models as a service, like when somebody build the models for you and the base model performance is a pretty shock. So. So it's like it's enable people to just like, okay, now I want to integrate AI into my product. I don't need to learn what gradient design is. Even though knowing that could really help. But yeah, it's like it makes the entry barrier really low for people who want to use AI to build product. And at the same time AI could capabilities are like so strong because it's also increased the possibilities like the type applications that AI can be used for. So I think yes, both entry barriers super low and I said demand for AI applications a lot bigger. So I feel this is very, very exciting. It opens up a whole new world of possibilities.

Yeah, it's like now you don't have the time, not even to spend time building this AI brain now you could just use it to do stuff. Such a, such an unlock. Okay, maybe just a final question. You get to see a lot of what's working, what's not working, where things are heading. I'm curious just if you have to think about in the next two or three years just where things are heading, what do you think? What do you think? How do you think building products will be different? How do you think companies working will be different? If you had to think of maybe the biggest change we expect to see in the next few years in terms of how companies work, I think in.

A lot of organizations they don't move that fast. Right. But at the same time they also move faster than I expected because again I think it's like bias, like I don't work with a dinosaur company, don't care. I think a lot of executive who come to me are like very forward looking. So maybe for me I'm very biased towards organizations. It's like move fast. I think one big change I see is in organizational structure. I think it's a lot of value place in. So before we have a lot of disjointed team, we have very clear engineering team, product team. But then there's a question of who should write eva, who should own the metrics. And it turns out it's like Ava, it's not a separate problem, it's a system problem. Right. Because you need to look into different components, how they interact with each other. You need to use the behaviors because you need to know what users care about so that you can, so that you can like write, write eval. Because it's like reflect what users care about. So all of that like you can sort it from like you look at your different component architectures, place guardrails and stuff. So it's just engineering, but understanding users is like what product. Right. So because of like a lot of things, any vault, extremely important. So like Zakat brings product team and engineering team, even marketing team, user acquisition very close to each other. So yeah, since in a way people are structuring so that more communications between previously very distinct functions. Another thing is I also see as teams of course think about what can be automated in the next few years and what cannot be automated. And I see that people already shedding actually it's a little bit like scary to think about it. But I also think it's like the team the web told me it's like okay this is good you and me but where we got rid of these functions right For a lot of things like previously outsourced for example traditionally it's a business outsourcing the smart core to them and can be done with not can be more systematized. So with that you can actually use AI to automate a lot of that. And so as a separation people think of more of what is the value of junior engineers or senior engineers. How should we structure engineering or for that so. So yeah so I do definitely think that is one thing to success organization people are just moving pieces around and like thinking about like use cases whether you wanted to like spin out new use cases and who would leads a new effort. And like yeah that is one big change. Another things in terms of like AI I think this is. I'm not sure how true this is. I guess I'm also on the camp of thinking that it has merit. It's a camp of okay base models we have probably not quite maxed out but we are unlikely to see really strong, crazy strong models. So you remember when we have GPT, right? And GPT2 which is a big step up an order of magnitude like better than like GPT and then GPT3 which like much, much bigger. GPT4 much much bigger and of course I mean GPT5 but like is GPT5 like that scale of like much bigger like a step jump compared to like the previous. I think it's a debatable. Right, so. So I think that it's like we have disappointment like the base model performance improvement is not going to be like mind blowing and it was in the last three years. So I think there's a lot of improvements. When I see in the post training phase, in the application building phase and yes also I think that's where I feel I would see a lot of improvement there. I was very interested in multimodality so we've seen a lot of text based but I think there's a lot of audio videos use cases that is very very exciting. And I think REO is not quite as solved as one thing because I do work with a couple of voice startups and when you talk to think about voice it's an entirely different beast. So let's Say you have chatbot, right? We go from a text chatbot to voice chatbot. It's like the concerns are completely different because now with voice chatbot, we need to think about latency because having multiple steps. First voice to text, text to text, and text question into text answer, and then text to voice answer, right? So it's like multiple hops and like latency become very important. And there's a question like, what does it make to sound natural? So for example, like people think like in AI and humans, when humans talk to each other, like if I say, I say you try to interrupt me and it's like chip, that's right. I would click pause and I try to hear you out, right? But sometimes I just even say say some word, like acknowledge. When I shouldn't stop, I just continue. So the question of like foreign interruption and whether it's like should I stop or not, like it's a big enough what perceived as like natural conversations. And there's also regulations, right? Because like a lot of time people want to build AI chatbot. Voice chatbots is sound like human, try to like trick users into thinking they're talking to humans. But it's of like maybe potential regulations saying like, okay, you have to disclose to users when you talk if the, if the bot is talking to is human or AI. So I think that's like just a whole space. I think it's not quite as sole as you think, but it's not quite like an AI foundation model problem, right? Because like a human interruption detection is actually a classical machining problem. Like it's a different framing of like you can build classifier for that or like the question of like, let us see a massive enduring challenge. Not an AI challenge. Of course it can be an AI challenge because we are trying to build a voice to voice model. So instead of having like having to first transcribe the voice from me into text and then get a model generous text answer and get another model to like transition speech. You can send your voice to voice directly. So that is something we're working on, but it's like very hard. Yeah, so. So, yeah, so like even audio, I think of it, it's like the easier than video, right? Because video have like both image and voice. It's already like pretty hard. So I think it's a lot of challenges in that space.

That was an awesome list of things. Let me mirror them back real quick. So what you're predicting in the next few years, things that will change in the way we work. And these actually resonate with so many conversations I've had on this podcast. So. So it's just kind of doubling down on where things are heading. One is the blurring of lines between different functions. Instead of just like engine design, engineering, everyone's going to be doing a lot of different things now. Two is just more of work being automated with agents and all these AI tools and just in theory, productivity going up. Third is shifting from pre training models to post training, fine tuning and things like that because SharePoint model models maybe are slowing down and how smart they're getting. Although I'll point folks to the chat with the co founder of Anthropic. He made a really good point here. He's like we're really bad at understanding what exponentials feel like. We're in the middle of that. And also models are being released more often so the difference between them we may not notice because they're just happening more often versus GPT3 came out like a year, I don't know after JPT2. So maybe true, maybe not. And then the fourth point you made is this idea of multimodal investing in multimodal experiences. I cannot wait for ChatGPT voice mode to get better at interruption. Like exactly what you're saying. I'm just like talking to it and then someone makes a little sound, it's like stop. Okay. And then you have to. And then it's like. And then it stops talking. So annoying.

I'm shocked that we don't have better voice assistant at home yet. I think I have been testing out a bunch. Honestly, I keep hoping oh my God, Zach would be the one. And then I don't know how many of them. I just like had to get away because they're not that good.

I think it's coming. I hear it's coming. Anthropic's working with someone that I don't know if it's launched or not yet.

Yeah. I sort of want to bring back to what you mentioned about like the guest like from Anthropic mentioned about the performance improvement. I think there's a big change. I think like this difference between a model based capability. So I'm talking about like the pre trained model. Right. Versus the perceived performance. So let's say it's just like I'm assuming it thought about like Are you familiar with the term in test time computer?

I don't think so.

Help us understand. So the idea is like okay, you have a fixed amount of compute, right? So you're going to spend a lot of compute on Pre tuning or tuning the model pre tuning and then I've spent a lot of some compute only 5 tuning and the ratio like pre tuning to push tuning compute is like crazy various different lab and also like Senzen has to spend compute on like generate inference when I have a trained and 5 10amodel and now you want to serve it to users. So I might type a question or prompt and it's like generate like do inference like and that requires a compute. And I guess I feel a discussion of like should I spend more compute on like pre tuning or fine tuning or inference. Right? Because like inference and people file. I was just like test time compute. So like spending more compute on inference is like called like test time like compute. Like the strategy of like just allocating more resources compute resource to generate inference when actually bring better performance. And how does that do it? Like let's say. Let's say you have a math questions, right? And maybe instead of just generic 1 Answer I can get like 4 different answers and say okay, whichever is the best according to some standard. Or like okay have four answers and then maybe like three of them say four a two and one of them says like 20 and it's like okay, three of them in agreement. So the answer should be 42. Right? So like just people shouldn't generate a bunch of it. Or another thing is like a lot of time like reasoning thinking is just like people shouldn't like generate more thinking tokens and spend more time thinking before showing the final answers. It's like require more compute but it's only giving more better performance. So yeah, so I think it's like from the user perspective, right? Like when the model spend more time exploring different potential answers, thinking longer it can give you much better final answers. But the base model itself does not change. Does that make sense?

Yes, that does. Absolutely. That is a good corollary to Ben Man's point Chip. We covered a lot of ground. I've gone through everything I was hoping to learn and more before we get to a very exciting lightning round. Is there anything else that you wanted to share? Anything else you want to leave listeners with?

So I do work with a few companies that does these things of like they want employees to come up with ideas. So there's a big debate on what is a better way for AI strategy. Should it be top down or bottom up? Should executive come up with one or two killer use case and everyone allocate resources to that? Or should you give engineers and PMs and smart people come up with ideas and I think it's just a mixture of both. So some companies, it was like, okay, we hire a bunch of smart people, let's see what they come up with. And they organize hackathons or internal challenge to get people to build product. And one thing that I noticed is a lot of people just don't know what you built. And it shocked me. I feel like we are in some kind of an idea crisis. Now we have all this really cool tool to have you do everything from scratch. I can have you design it, can have you write code, you can have build websites. In theory we should see a lot more. But at the same time people are like somehow stuck, like they don't know what to build. And, and I think it's like maybe a lot of had to do with like, maybe like society expectations because like we have gone through, we have gone into this phase of like specializations, like people like very highly specialized and people are supposed to do like focus on one thing really well instead of being a big picture and we don't have a big picture of you. It's hard to come up with like ideas of what you build. So, so I know what like when I work company on this hackathon, like we do work out like how come up with a guideline like how to come up with ideas. And usually what we think of is like, okay, like one tip is like go look from the last week, right? Like for a week just like pay attention to what you do and what frustrates you. And when something frustrates you, think like, is there anything we can do? Is there like can it be done a different way so it's not frustrating and you can talk like people can swap accept sub no teams. And I even see they come on frustrations. Maybe just something you can think about is just to build something around that. So yeah, so I feel like just like notice like how we work thinking of like ways to constantly ask questions like how can this be better? And then I just build something to like address the frustrations. I think it's a good way to just like learn and adopt AI.

I think people have felt exactly what you're describing every time they open up one of these vibe coding tools where they could just describe anything you want. I'm like, I don't know, what do I want? And I love this very tactical piece of advice. Just like what frustrates you? Just pay attention to where you're frustrated. For example, I just built a very cool little vibe coded app. I was working on a newsletter post inside Google Docs and I, I pasted all these images into the Google Doc from screenshots and stuff and. And then I forgot. Oh yeah, you can't take images out of Google Docs. It's like this hotel California experience where you can paste stuff into it. Very hard to get images back out. So I just went to all the vive coded tools and just build an app that I can give you a Google Doc URL and it's let me download all the images automatically and it worked amazingly well and I made it really cute and I'll link to it in the show notes.

Oh, I would love to say that I'm very bullish on using AI. Just create like micro tools. It's just something just like make your.

Life a bit easier 100%. I feel like that's one of the main ways people are using these tools. Just like a little niche problem they have with that chip. We've reached our very exciting lightning round. I've got five questions for you. Are you ready?

Yeah, always. No, no, no. It depends on how hard the questions are.

They're very consistent across every guest, so I imagine you've heard them before. First question, what are two or three books that you find yourself recommending most to other people?

Oof. I'm really terrified of like book recommendations because I feel like what books a person should read really depends on what they want and where they're in life and where they want to get to. But I just several books that I do think is actually really changed the way I think and see the world. So one thing is a selfish gene that's like to understand it actually changed. It actually helped me with the question like whether I want to have kids or not. Because it's like understanding more of like. Yeah. A lot of our functions, the way we operate is functions of our genes and genes. One to the one thing was like to procreate. So. So yes, in a little way. But also like the book also propose another thing is like so everyone wants to live forever, right. And maybe it's not like consciously but subconsciously we do, we do want that. And I said two ways. Like one is like via genes, like genes. One's just like going to continue forever but it's also there through ideas. I think there's something going meme. It's like being a boss if you have some ideas out there and then it's like last for a long time. That's on voice to like live on. I know it's like it's a little bit like abstract, but I thought it's very interesting. The other books I really, really like is like from like the book from Singaporean previous. I think he says the father of Singapore. I don't know. Like, I'm not sure what's the title it, but like he did so he was the one who left Singapore from. He's changed Singapore from a third world country to a first world country within 25 years. And I have never seen any country leaders spend so much effort into like putting down his thought on like how to build a country. Like. Like that. Yeah. As I talk a lot about like public policy, like how to like create policies that encourage people to do the right things that is good for the nations and also talking about like foreign affairs, foreign policies, like the liberation of like the country with other. So it's a really good book to think about. For me. It's a system thinking, but like it's a different kind of system which is country which a lot of us don't get a chance to like ever experiment in our life. So it's good to learn about that.

What was the name of that second book?

It's called like From Third to First World Class. I think I have it somewhere here.

Yeah, there it is. Show and Tell. That's. That's awesome. I definitely want to read that. That's a really good tip. I've heard a lot about just the impact he's had and I've seen all these videos on Twitter of just his really wise insights into how to build a thriving society. And clearly it works.

How does he have time to write such a thick book? It's like insane.

That is. Claude, please summarize. I'm just joking. By the way, Selfish Gene, I also absolutely love that book. That is such a good choice. It's such an under the radar kind of book that really changed the way I see the world as well. So really good pick. Okay, next question. Do you have a favorite recent movie or TV show you really enjoyed?

So I watch a lot of movie and TV shows as a research because I I working on my first novel and I recently sold it. So I'm interesting. Like what makes it. It's a drama, it's not a science fiction or anything that like tech people usually read. So it's very like. I know it's a very out of the left field of left field and like very so. So it's like reading, watching TV to see like what kind of stories become popular, trying to understand the trope and stuff like that. So I'm not sure if the aud.

Well, what's one. What's one that taught you something about writing?

I think it, like, Yamsi palace, it's a Chinese TV show.

Cool. Okay. I haven't heard that one on the podcast before. Okay, cool. Next question. Do you have a life motto that you often think about, come back to when you're dealing with something hard, whether it's in work or in life.

This sounds very nihilist. I think the same, like, in the end, nothing really matters. Usually I think of, like, in the grand scale of things, like, in a billion years, nothing will. Like, no one will ever be there. I think, okay, someone will argue with me about that. So I go to say, like, so my theory is like, in a billion years, like, none of us will never exist. Like, so, like, whatever. Like, messy things, like, crazy things we do or, like, how bad we do it. I mean, no one would be remember. Wouldn't be there to remember it. And I think in a way it's like, it sounds scary, but it's very liberating because it just allows me, okay, let's just try things out, right? Like, why does it matter? And there's a story of, like, recently. So we have some family member who passed away recently. And I was talking to my dad because I couldn't be home for that. I was asking my dad, like, okay, say anything I can do to make the person, like, something like, comfort. So anything you can get that person. And my dad was just like, what can he possibly want at this moment? Like. And this made me feel like, at the end of life, like, there's nothing that can bring you, like. Like, material can bring you joy. There's no, like, money, no product, nothing. And in a way to make feeling okay, what really, do I really care about at the end of the day? So I guess it's like a thing about it. It's just like, okay, maybe I fail it. Maybe I don't get a contract. Maybe those things, like, at the end of life, like, I don't think that actually really matters. So in a way, just like, it's quite liberating.

I know you said it might be nihilistic. This is what Steve Jobs shared too, in one of his most famous speeches. Just, we will all die someday, so don't take things so seriously. And it is freeing. Absolutely. It just makes you appreciate every moment, every day you have. Just like, yeah, let's just do something hard and scary. Okay, final question. You talked about how you're writing a novel. Most people in Tech have never written something creative and fiction. What's just like one thing you learned in the process about how to write better stories, better fiction.

A lot of time when we read, we get tripped up by some small things. So I think I want to do creative writing because I just want to go a better writer. And it tells us maybe try my. A different audience could have me become better at anticipating what this different type of audience would want to hear and what they care about. So this is a way for me to get a. So I think about writing even any kind of content creations. It's about predicting the user's reactions, right?

The next token. Just kidding.

Yeah. So like you do a podcast, it's like, okay, what kind of things that the users could find engaging, right? And I find it's like a little bit like. And a lot of companies, like you have like launch a product, you have a narrative coming out, say, okay, what guide? How do we position this product in a way that like users would want, right? So I feel like I have done technical writing for a while and it's felt like I have had some experience like trying to predict what engineers would want to hear or like care about. But then I don't have an experience like this completely different type of audience. So that's what I want to do, like creative writing, writing story. And that's why I was doing a lot of research on the question. I mean, going to research, I actually enjoyed a lot, like watching a lot of dramas. I just see like what people like. So one thing that I care about is just like, I think I learned it's like for like emotional journey was from an editor, right? So like when we write something, like we care about like how users would feel like across the story, like we want something in the beginning, right? We want something just like we need to have a hook so that people continue reading. But we also don't want too much of like drama because we'll get like too tired, right? Like, because like you're emotionally exhausted. Like, because it's like you're being like emotionally manipulated, like a lot of times. So if you have like emotional, emotional journey, maybe have like some, some climax or like some, something more chill, like maybe like and so care about. Another thing that I didn't realize, like, for me, for technical writing, you entirely focus on the content. Like the argument. It's very impersonal, right? Like it like, for example, like people like ML compilers, like, doesn't matter if they like the person telling them about compiler or not, right? Because it's just like, objective, like, but like, for a novel, people care about, like, character likability. So. So, like, in the first version is my story and makes the characters, like, a little bit more, like, very, very logical, very rational, and just does everything just like, very rationally. And then the feedback I got is, I have a very good friend. And he was. He's an amazing person. He's a great person. And he was like, chip, I'll be honest, I hate that person. So it doesn't matter as a story. It's just like, the person is so unlikable to say he doesn't want you continue. So it's a second version. It makes a person, the character a lot more likable. Like, what. How she makes a character more likable is that you put in some vulnerability, like, some time. Okay. Maybe it's a person, like, has setback because somehow we can relate to it. See, in a lot of ways, like, it's very interesting. It's like a lot of it is, like, yeah, a lot of it is. It's about, like, understand the emotional bit, like, how the users feel not just about the story, but also about the characters.

That is so interesting. Wow, I learned a lot more there than I thought. That was awesome. Really good example, Chip. Two final questions. Where can folks find you online if they want to reach out and maybe work with you or maybe even just share the stuff that you offer if folks want to reach out. And then how can listeners be useful to you?

I'm like, I'm on social media, LinkedIn, Twitter. I don't post a lot, but I keep telling myself that I should do more because I quite like the convention with. With. With. With readers. So I'm sure you're about to start a slight a subspec. So I have, like, a placeholder for subspec right now. And I'm thinking of doing it for my system thinking because I think it's a very interesting skill. And so, like, thinking of doing a YouTube channel on book reviews and basically books that help you think better. So I think it's. The first book I'm going to review is probably like, this book because it's like my favorite book growing up and I have been like, keep on reading it. So, yeah, so how can it be helpful? Like, send me books that you like, books that help you have changed the way you think or change you the way you do anything. So I would appreciate it.

Amazing. I'm excited to read that book. Chip, thank you so much for being here.

Thank you so much Lenny for having me here.

Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show@lennyspodcast.com See you in the next episode.