OpenAI researcher on why soft skills are the future of work | Karina Nguyen (Research at OpenAI, ex-Anthropic)

Not only are you working at the cutting edge of AI and LLMs, you're actually building the cutting edge.

When I first came to Andarab, I was like, oh, that I really love frontend engineering. And then the reason why I switched to research is because I realized, oh my God, Cloud is getting better at frontend. Cloud is getting better at coding. I think Cloud can develop new apps.

What skills do you think will be most valuable going forward for product teams in particular?

Creative thinking. You kind of want to generate a bunch of ideas and filter through them in order to the best product experience. I think it's actually really, really hard to teach the model how to be aesthetic, or really good visual design, or how to be extremely creative in the way they write.

What do you think people most misunderstand about how models are created?

When you taught the model some of the self-knowledge of you actually don't have a physical body to operate in the physical world, the model would get extremely confused.

Today, my guest is Karina Nguyen. Karina is an AI researcher at OpenAI, where she helped build Canvas, Tasks, the 01 Chain of Thought model, and more. Prior to OpenAI, she was at Anthropic, where she led work on post-training and evaluation for the CLaUd-3 models, built a document upload feature with 100K context windows, and so much more. She was also an engineer at New York Times, was a designer at Dropbox and at Square. It's very rare to get a glimpse into how someone working on the bleeding edge of AI and LLMs operates, and how they think about where things are heading. In our conversation, we talk about how teams at OpenAI operate and build products, what skills she thinks you should be building as AI gets smarter, how models are created, why synthetic data will allow models to keep getting smarter, and why she moved from engineering to research after realizing how good LLMs are going to be at coding. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It's the best way to avoid missing future episodes, and it helps the podcast tremendously. With that, I bring you Karina Nguyen. This episode is brought to you by Interpret. Interpret unifies all your customer interactions from Gong calls to Zendesk tickets to Twitter threads to App Store reviews and makes it available for analysis. It's trusted by leading product orgs like Canva, Notion, Loom, Linear, Monday.com, and Strava to bring the voice of the customer into the product development process, helping you build best-in-class products faster. What makes Interpret special is its ability to build and update customer-specific AI models that provide the most granular and accurate insights into your business, connect customer insights to revenue and operational data in your CRM or data warehouse to map the business impact of each customer need and prioritize confidently, and empower your entire team to easily take action on use cases like win-loss analysis, critical bug detection, and identifying drivers of churn. with interprets AI assistant wisdom. Looking to automate your feedback loops and prioritize your roadmap with confidence like Notion, Canva and Linear. Visit bnterpret.com/lenny to connect with the team and get two free months when you sign up for an annual plan. This is a limited time offer. That's interpret.com/lenny. This episode is brought to you by Vanta and I am very excited to have Christina Cassioppo, CEO and co-founder of Vanta joining me for this Very short conversation.

Great to be here. Big fan of the podcast and the newsletter.

Vanta is a long time sponsor of the show, but for some of our newer listeners, what does Vanta do and who is it for?

Sure. So we started Vanta in 2018, focused on founders, helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications like SOC 2 or ISO 27001. Today, we currently help over 9,000 companies, including some startup household names like Atlassian, Ramp, and LangChain, start and scale their security programs and ultimately build trust by automating compliance, centralizing GRC, and accelerating security reviews.

That is awesome. I know from experience that these things take a lot of time and a lot of resources, and nobody wants to spend time doing this.

That is very much our experience, but before the company and some extent during it. But the idea is with automation, with AI, with software, we are helping customers build trust with prospects and customers in an efficient way. And you know our joke, we started this compliance company so you don't have to.

We appreciate you for doing that. And you have a special discount for listeners. They can get $1,000 off Vanta at vanta.com/lenny that's vanta.com/lenny for $1,000 off Vanta. Thanks for that, Christina. Thank you. Karina, thank you so much for being here. Welcome to the podcast.

Thank you so much, Lenny, for inviting me.

I'm very excited to have you here because not only are you working at the cutting edge of AI and LLMs, you're actually building the cutting edge of AI and LLMs. You recently launched this feature, which basically the first agent feature of OpenAI. I also just did this survey. I don't know if you know about this. I did this, I did a survey of my readers and asked them what tools do you use every day in your work and most use. And ChatGPT was number one above Gmail, above Slack, above anything else. 90% of people said they use ChatGPT regularly. It's absurd. It wasn't around two years ago. Yeah. Also, we're recording this the week that OpenAI announced Stargate, which is this half trillion dollar investment in AI infrastructure. So there's just like a lot happening. constantly in AI and you have a really unique glimpse into how things are working, where things are going, how work gets done. So I have a lot of questions for you. I want to talk about how you operate and how you work at OpenAI, where you think things are going, what skills are going to matter more and less in the future, and also just where things are going broadly. So how does that sound?

Sounds great. Thank you so much. Yeah, I was extremely lucky. to join early days on DALL-E and kind of learned a lot of things there. And I joined OpenAI around like eight months ago. So yeah, I'm excited to dive more into that.

Okay, I'm gonna definitely ask you about the differences between those, but I wanna start more technical and just dive right in. I wanna talk about model training. People always hear about models being trained, these big models, how much data it takes, how long it takes, how much money you toss. It takes how we're running out of data, which I want to talk about. Let me just ask you this question. What do you think people most misunderstand about how models are created?

Model training is more an art than a science. And in a lot of ways, we as model trainers think a lot about data quality. It's one of the most important things in model training is how do you ensure the highest quality data for certain interaction model behavior that you want to create. But the way you debug models is actually very similar the way you debug software. So one of the things that I've learned early days at Anthropic was we've discovered, especially with Claude 3 training, when you taught the model some of the self-knowledge of, hey, you actually don't have a physical body to operate in the physical world, But then at the same time, we had data that taught the model some of the function calls, which is like, this is how you set the alarm. And so the model would get extremely confused about whether it can set an alarm, but it doesn't have a body in the physical world. So it's like the model gets confused and sometimes it's over-refused. So sometimes it says, I don't know, sorry, I cannot help you. And so there is always a balanced trade-off between how do you make the model to be more helpful for users, but also not being harmful in other scenarios. So it's always about how do you make the model more robust and operate across a variety of diverse scenarios.

That is so funny. I never thought about that. Most of the data that it's trained on is kind of assuming it's a human describing the world and how they operate, and there's Assumes there's a body and you could do things in the model told you don't have a body.

Yeah.

Okay. I want to talk a little bit about data while we're on this topic. I know you have strong opinions here. There's kind of this meme that models are going to stop getting smarter because they're running out of data. They're trained in a large part on the internet, and there's only one internet, and they've already been trained on it. What more can you show them about the world? And there's this trend of synthetic data, this term synthetic data. What is synthetic data? Why do you think it's important? Do you think it's going to work?

I think there are two questions here we can unpack one at a time. But people say we're hitting the data wall. I think people think more in the terms of pre-trained large models that are trained on the entire internet to predict the next token. But what actually the model is learning during that process is actually how do you compress the compression algorithm here? The model learns to compress a lot of knowledge and it learns how to model the world. So the next prediction of the word, teach me how to drive, basically, and you only have a few words that will match that, a car. So the model actually learns about the world in itself. So it's like it's modeling human behavior, sometimes it's modeling. And when you talk to like pre-trained models, which are very, very large, they're actually extremely diverse and extremely creative because you can talk to almost any Reddit user through pre-trained model. But I think what's happening right now was like new paradigm of like 01 series is that like, the scaling in post-training itself is not hitting the wall. And that's because basically we went from raw data sets from pre-training models to infinite amount of tasks that you can teach the model in the post-training world via reinforcement learning. So any task, for example, how to search the web, how to use the computer, how to write well, all sorts of tasks that you're trying to teach the model, all the different skills. And that's why I'm saying there's no data wall or whatever, because there will be infinite amount of tasks. And that's how the model becomes extremely super intelligent. And we are actually getting saturated in all benchmarks. So I think the bottleneck is actually in evaluations that we don't have.

All.

The frontier, like EVAS, like, I don't know, GPQA, which is a Google-prove question answering, PhD level intelligence. The benchmark is getting to, I don't know, more than 60-70%, which is what PhD gets. So it's like, we're literally hitting the wall in evolves.

I want to follow both those threads. So the first is on this idea of synthetic data. Is a simple way to understand it that the models are generating the data that future models are trained on, and you ask it to generate all these ways of doing stuff, all these tasks, as you described, and then the newer models trained on this data that the previous model generated.

Some tasks are synthetically curated. So this is an active research area. How can you synthetically construct new tasks model to learn? Sometimes, when you develop products, you get a lot of data from the product and use feedback and you can use that data too in this post-training world. Sometimes you still want to use human data because actually some of the tasks can be really, really hard to teach. Experts, only know certain knowledge about some chemicals or biological knowledge. So you actually need to tap into the expert knowledge a lot. So, yeah, I think to me synthetic data training is more for product, it's a rapid model iteration for similar product outcomes. And we can dive more into it, but the way we made Canvas and tasks and new product features for ChatGPT was mostly down with vice and that of training.

Let's actually get into that. That's really interesting. I want to talk about evals, but let's follow that thread. So talk about how this helped you create Canvas.

So when I first came to OpenAI, I really had this idea of like, okay, like it would be really cool for ChatGPT to actually like change the visual interface, but also change like the way it is with people. So going from being a chatbot to more of a collaborative agent and a collaborator is like a step towards more ganted existence that become innovators ultimately. And so the entire team of applied engineers, designers, products, research kind of got formed in the air almost out of nothing. It's just a collection of people who just got together and we rapidly started iterating with each other. Actually, Canvas is one of the, I would say, the first project at OpenAI where researchers and applying engineers started working together from the very beginning of the product development cycle. And I think there's a lot of things that we have learned on the way. But I definitely came with the mindset of we need to do a really rapid model iteration such that it would be much easier for engineers to work with the latest model possible, but also learn from user feedback or early internal dogfood, how do we improve the model very rapidly? And it's really hard to figure out how people, when you deploy a product, how people would be able to use it. And so the way you synthetically train the model is basically figuring out what are the most core behaviors that you want this product feature to do. And for Canvas, for example, it came down to three main behaviors. It was how do you trigger Canvas for prompts like write me a long essay when the user intention is mostly iterating over long documents, or write me a piece of code, or when to not trigger Canvas for prompts like, can you tell me more about precedent? I don't know, some of the general questions. So you don't want to trigger Canvas because the user intention is mostly getting answer and not necessarily iterate over the long document. The second behavior is How do we teach the model to update the document when the user asks? So one of the behaviors that the model is actually have some agency on autonomy to literally go to the document and select specific sections and either delete it or edit, so highlighted and rewrite certain sections. So sometimes the model Sometimes a user would just say, change the second paragraph to be something friendlier. And we would have to teach the model to literally find the second paragraph in the document and change it to a friendly tone. So basically you teach both how to trigger edit itself, but also how do you teach the model to get higher quality edit for that document. In case of coding, for example, And there's also the question of how good the model is at completely rewriting the document versus having very specific targeted edits. So that's another layer of decision boundary within edit itself. It's select the entire document and rewrite completely, or you want to have very targeted custom behavior. And when we first launched the model, we would bias the model towards more rewrites. because we thought the quality of the reviews were much higher. But over time, you're shifting based on user feedback and what they're learning from iterative deployment. Lastly, the third behavior that we taught synthetically, the model is how to make comments on any document. So the way we used it is we would use a one model to produce, to simulate user conversation, let's say, write me a document about XYZ. Then we used a one to produce the document. And then we injected user prompts to be like, oh, make some comments, critique my piece of writing, or critique this piece of writing that you just made. And then we taught the model to make comments on the document. on very specific targeted documents. This is also what kind of comments you want the model to make. Do they make sense or not? How do you teach the quality of that? And it all came down to measuring progress via very robust evals. But yeah, this is how you used a long kind of synthetic data generation for training.

Okay, that's so interesting. So you talk about this idea of teaching the model and you mentioned how it's using synthetic data to teach the model different behaviors is a simple way to think about it. Basically, that's where you do that by showing it what success looks like using basically evals. Is that the simple way to think about it? Like, here's what you doing this successfully would look like, and that teaches it. Okay, I see.

This is what I should do. Yeah, great.

Yeah. Amazing. Yeah, you got it. Okay, got it. I want to start unpacking what your day-to-day looks like as you're building these sort of things. Is it like you sitting there talking to some version of ChatGPT crafting these evals.

Sometimes I do that, but sometimes I do set for ChatGPT. Actually, I think I learned this so much from Andrew Ng. It's like people spend so much time just like prompting models and quality in a little bit bash all the time, and you actually get a lot of new ideas. How do you make the model better. It was like, oh, this response is kind of weird. Why is it doing this? And you start debugging or something, or you start figuring out new methods, or how do you teach the model to respond in a different way, have better personality, let's say. So it's the same thing of how personality is made in the models of the models, very similar methods. But yes, I think my time I think when they first came, I was mostly research IC work. So I was building a lot of, I was writing code, changing models, writing evals, working with PMs and designers to learn, teach them how to even think about evaluations. I think that was a really cool experience and I think this is an adoption of how do we do this product management of AI features or AI models? Yeah, but now it's mostly management and mentorship. I'm still doing I see research code after 4:00 PM, although, but yeah, I just kind of changed.

All right, don't talk too much about being a manager, because everyone's firing their managers. Who needs managers anymore? That's what I hear now. Just kidding. It's interesting that so much of your time was spent on teaching product teams how evals integrate and how important that is. And I've heard this a few times and I haven't personally experienced it yet, so I think it's an important thread to follow is just how writing these evaluations is going to become increasingly an important part of the job of product teams, especially when they're building AI features and working with LLMs. So can you just talk a bit more about what that looks like? Is it like sitting there with an Excel spreadsheet? basically showing like here's the input, here's the output, here's how good the result was. Talk about what that actually looks like very practically.

It certainly depends on what you're developing, but there are various types of evaluations. So sometimes I do ask product managers or there's also new roles that we have like model designers to go through some of the user feedback maybe or think of various user conversations. that should have triggered, under these circumstances, it should trigger canvas. And then you have this ground truth label of, okay, with this conversation, it should trigger canvas. Under this conversation, it should not trigger canvas. And you have this very deterministic, kind of eval, that for decision boundary behaviors, it's like this. When we were launching tasks, for example, how do you make correct schedules is actually really hard for the model. But we bolted out some of the deterministic evaluations that is like, okay, if the user says 7 p.m., the model should say 7 p.m. So you can have deterministic evals whether it's pass or fail. So yeah, and the way it works is sometimes I ask product managers to just go create a Google sheet, have different tabs, and what's the current behavior, what's the ideal behavior, and why or some notes. And sometimes we usually use it for eval, sometimes we use it for training. Because if you give the spreadsheet to a one model, it can probably figure out how to teach itself a good behavior. And I think there are second type of evals that is kind of more prevalent is human evaluations. And you can have specific trainers or you can have internal people to... When you have a conversation of the prompt and then you have various completion of models, you choose the win rate. Which model is the best? Which model produced the highest quality comment or edit? And then you can have continuous win rates. And as you develop new models, it should always win over the previous models. So it depends on what you want to measure.

So interesting. Basically what I'm hearing and this is something I'm learning about as I talk to people is product development might move from this, here's a spec PRD, let's build it together, and then cool, let's review it, are we happy with this? From that to, hey, AI, build this thing for me. And here's what correct looks like. And I'm spending all my time on what does correct look like on evals essentially.

You definitely want to measure progress of your model. And this is where evals is, is because you can have prompted model as a baseline already. And the most robust evals is the one where prompted baselines get the lowest score or something. And then because then you know, if you trained a good model, then it should just he'll climb on that eval all the time while not also regressing on other intelligence evals. So I think it's more what that's what I'm saying. It's more of an art than science. It's like, okay, if you optimize the model for this behavior, you kind of don't want to bring damage in other areas of intelligence. And this is happening all the time in every lab, in every research team. I would say, Prompting is also a way to prototype new product ideas. Early days at Andarabek, when I was working on file uploads feature, I remember I was just prompting the model to just launch in 100 key contexts. I was just prototyping this in my local person. I did the demo, people really, really loved it, and they just wanted API for file uploads or something. And then that's when it clicked to me. I also wrote a blog post a long time ago. It clicked to me that prompting is a new way of product development or prototyping for designers and for product managers. For example, one of the features that I wanted to do is have personalized startup prompts. So whenever you come to Claude, it should recommend you starter prompts based on what your interests are. And so you can literally do it prompting for that.

Experiment with that.

Another feature was generating titles for the conversations. It's a very small micro experience, but I'm really proud of. The way we did that was we took five latest conversations, from the user, like ask the model, like what's the style of the user? And then like for the next kind of new conversation, the generated title will be of the same style. It just has like really little like micro experiences like this. That's so cool.

Did you do that at Anthropic or at OpenAI? At Anthropic. Okay, cool. I love the file upload feature that Claude has, by the way. Oh, ChatGPT doesn't have that yet, is that right?

I think it has. I think like the way it's implemented.

Is like very different though. Okay, maybe it's the PDF feature. Cause I use it all the time with Claude. Okay. That's cool. Someone needs to get on that. Man, it's wild how many features you built that I use every day and that many people use every day. This prototyping point you made is really important. It's something that comes up a ton on this podcast also of how that is maybe the way that AI has most impacted the job of product builders recently is just prototyping instead of going from showing just like, here's a PRD, here's a design, PMs more and more just, here's the prototype, the idea that.

I have and it's working, you can play with it. Yeah.

Okay. I want to spend a little more time on how you operate. So you talked about you built this in Launchlist Tasks feature. Is that the way you describe your tasks? Yeah. So talk about how that emerged and let's better understand just how you collaborate with product teams and how OpenAI works in that way, whatever you can share there.

I think Canvas and Tasks are going into the bucket of our projects where it's more like short or medium terms. And actually the way Canvas and Tasks came about to be was it started as one-person prototyping and creating a spec. It's kind of like PRD. It's creating a spec of the behavior of the model. I don't think Tasks is extremely groundbreaking feature necessarily. What makes it really cool is because the models are so general, the model can now search, they can write sci-fi stories, they can search for stocks, they can summarize the news every day because the models are so general, giving something familiar to people that notification is very familiar, having reminders is very familiar. Creating a form factor for the people who are very familiar, same as Canvas, right? It's like Google Docs is very familiar. But then you add magical AI moment and it becomes very powerful. But the way it comes, usually operationally, it starts as a prototype, literally prompted prototype of how you would want the model to behave. For tasks, for example, you need to design I literally like design systems design thinking is like, okay, well, if the user says, remind me to go to lunch at 8:00 AM tomorrow, okay, what kind of information does a model need to extract from that prompt in order to create a reminder? And so this is how you design a spec for a new feature, like a tool, Canvas and Tasks are all tools. So it's like, how do you create the tool spec? And then it's mostly developing JSON schema. I was like, okay, from this prompt, maybe the model should extract the time that the user requested. And then you think about which format do you want the time to be? And then how do you want the model to notifiy you? Basically, the user should give instruction to the model. And then this instruction would fire off every day or something at that particular time. So for example, if you say search, every day I want to learn now about the latest AI news, the model should rewrite into, okay, search for the latest AI news, and this task will get fired at that particular type of the model that the user requested. and then, you know, your design was like tools back, and then actually, I don't know, I feel like sometimes, like, it's like through conversations, I either like people ask me to join the existing team and they're like, oh my God, we need researchers, or like, we need some support, we need to train the models. Or sometimes, like, with canvas, it's like mostly, like, I just pitched the idea of, like, It got staffed quite immediately during the break. So I feel like it depends on the project. And usually with staffing, it's mostly like a product manager, model designer, actual product designer, a couple of researchers on a project like applied engineers. Depends on the complexity of the project. And then, you know, for tasks it took like, I don't know, like two months or so. to go from zero to one basically.

Oh wow.

For Canvas, this was like four, five months, I guess, to build from zero to one. But yeah, and then you teach product managers how to build evals and maybe, how do we not only ship the beta feature, but how do we think longer term? What kind of cool features did you want tasks? to have, I think it would be nice for tasks to be a little bit more personalized. It'd be nice to have to create tasks via voice on a mobile, right? So you kind of need to, this is how you get research roadmap right here is thinking how the future will be developed in the future. And then from there on, it's like you start creating datasets with EVAZ, you want to make sure that goes Well, and then you need to have a trade-off between what methods you want to use. And the reason why I really love relying purely on synthetic data instead of collecting data from humans is because it's much more scalable. It's cheap, less than how you literally sample from the model and you teach the core behaviors of the models, and that will generalize to all sorts of things. diverse coverage. And when you launch the beta feature, you learn so much from the users that you can, like, all your synthetic sets can be shifted in the distribution of how the users behave on the product behavior, and this is how you improve. And this is what happens in Canvas too, when we learn from beta to GA.

This episode is brought to you by Loom. Loom lets you record your screen, your camera, and your voice to share video messages easily. Record a loom and send it out with just a link to gather feedback, add context, or share an update. So now you can delete that novel-length email that you were writing. Instead, you can record your screen and share your message faster. Loom can help you have fewer meetings and make the meetings that you do have much more productive. Meetings start with everyone on the same page and and early. Problem solved, Time saved. We know that everyone isn't a one-take wonder when it comes to recording videos, so Loom comes with easy editing and AI features to help you record once and get back to the work that counts. Save time, align your team, stay connected, and get more done with Loom. Now part of Atlassian, the makers of Jira. Try Loom for free today at loom.com/lenny that's L-O-O-M.com/lenny. / Lenny. Something that I want to help people understand, and I don't even a hundred percent understand this, is what's the simplest way to understand the job of a researcher versus, say, a model designer and other folks involved? Like, what's the simplest way to understand.

What researchers do at OpenAI? So the project that I described, I mostly like product oriented, like research is mostly like product research. Another part component of my team is actually more like longer term exploratory projects. and it's more about developing new methods, understanding those methods, and a variety of circumstances. So basically develop new methods, you need to follow very similar recipe of building evals, but it's more sophisticated evals. You want to have auto distribution, or if you want to measure generalization, If I don't need to like after that. But it's basically more science in a way where, you know, if we talk about synthetic data, like one of the hardest things about synthetic data is like, how do you make it like more diverse? Diversity in synthetic data is like one of the most important questions right now. And it's just like exploring ways to inject diversity as a general method that will work for all is like a one of the research explorations. other ones is more like developing new capabilities. I feel like it's always about you work on this new method and you have signs of life that it's working. Either you think of how do you make it more general or you think of how do you make it very useful? And this is how longer term projects become more medium to short term projects.

That makes sense. Essentially working on developing ways to make the models smarter. 04 or 05 or 06. New ways to, like 01 was a big breakthrough, right? The way it operates where it's not just here's your answer, it actually thinks and has takes time to think through the process of coming up with an answer. Okay. Very helpful. Speaking of that, of thinking about the future, where things are going, I want to spend some time on just this insight that basically you are building the cutting edge of AI. like at the very bleeding edge of where AI is going and where it is. And so I'm very curious to hear just your take on how you think things are going to change in the world and how people work based on where you see things are going. And I know it's a broad question, but let's say like in the next three years, how do you see the world changing? How do you see people's way of working changing?

It's a very humbling experience to be in both labs, I guess, like to me, when I first came to Andarab, I was like, oh, that I really love frontend engineering. And then, like, the reason why I switched to, like, research is because I realized at that time, it's like, oh, my God, like, cloud is getting better at, like, front end. Like, cloud is getting better, like, coding. I think cloud can, like, develop new apps or something, and so, like, it can, like, develop new features for the thing that I'm working. So it's like, it was kind of like this meta realization where it's like, Oh my God, like, the world is actually changing. And they're like, when we first launched 100k context at that time, obviously, you know, I'm thinking about like, from factors that's like, yeah, like file uploads were like very natural, very familiar to people, but you could imagine we could just like make like infinite chats in the cloud.ai app, right? Like as if like it's like in 100k context. But because file uploads, it's like foreign followers function, it's like the form factor of the file uploads kind of enable people to just literally upload anything, the books, any reports, financial, and ask any task to the model. And then I remember it was like enterprise customers, financial customers are really interested in that. It's like, Oh, wow. It's actually one of the very common tasks that people do in that setting. It was kind of crazy to see how some of the redundant tasks are getting automated basically by these smart models. And they are entering the era where I actually don't know, for example, sometimes if L1 gives me the correct answer, or not, because I'm not an expert in that field. And it's like, I don't even know how to verify the outputs of the models. It's because all my experts know they can verify those. So, yes, so basically there are trends that are going on. The first trend is the cost of reasoning and intelligence is drastically going down. I had a blog posts about this. Maybe I should update them latest benchmarks because at that time, everybody was doing one benchmark and they'd be quickly saturating the benchmarks. And now we need to do the same plot but with another frontier eval. But the cost of intelligence is going down because it becomes much cheaper. Small models are becoming It was smarter than large models, and that's because of the distillation research. This happened with Claude 2. I was working on Pustak 2, Claude 2, and I realized it was much smarter than Claude 2, which was way bigger, something like that. But the power of small models become very intelligent and fast. and cheap, we are moving towards that world. That has multiple implications, but that means that people will have more access to AI, and that's really good. Builders and developers will have much better access to AI, but also it means all the work that has been bottlenecked by intelligence will be unblocked. So anyone I'm thinking about healthcare, right? If I have instead of going to a doctor, I can ask ChatGPT or give ChatGPT a list of symptoms and ask me, which would I have a cold, flu, or something else? I can literally get the access to doctor almost. And there's been some research studies around that.

Yeah, there was a New York Times story about that where they compared doctors to doctors using ChatGPT to just ChatGPT and just ChatGPT was the best of them all. Like, doctors made it worse.

Yeah, that's crazy. Like, right? Like education, I think I would have dreamed if I had the tool like ChatGPT when I was young and could learn so much, but it's like people can now learn almost anything. from these models so they can learn new language, they can learn how to build new lookups, like, I don't know, anything that you want. And like, I'm so, like, it's humbling to like have like launch canvas and like bring that thing to the people, enable them to do something else that they couldn't have ever before. And I think this is, there's something like magical around this experience. So education has will have massive implications like, I guess, like scientific research, right? I think it's like the dream of any AI research is like augment AI research. It's kind of scary, I would say, which makes me think that people management will stay. It's one of the hardest things to it's like emotional intelligence for the models. Creativity in itself is one of the hardest things. So writers, I don't think People should be worried as much. I think it's like, I think it's alleviated a lot of like redundant tasks for people.

This is awesome. Okay, I want to follow this thread for sure. And it's funny that what you described is like, you were an engineer at Anthropic and you're like, okay, Claude is going to be very good at engineering. This isn't going to be a potentially career long term, so I'm going to move into research and AI is going.

To need me for a long time to build it, to make it smarter. I would say we still have, I think Canvas team has still have really cool front engineers that are really like, people who really care about interaction design, interaction experiences. I don't think models are there yet, but we can get the models to this top 1% of front end or something. For sure.

So what I want to move on to next along these lines is just, and this is just speculation, But what skills do you think will be most valuable going forward for product teams in particular? So folks are listening and they're like, okay, this is scary. What should I be building now to help me stay ahead and not be in trouble down the road? What skills do you think are gonna be most, more and more important to build?

Yeah, I think like creative thinking, You kind of want to generate a bunch of ideas and filter through them in order to build the best product experience. You want to build something that the most general model will not replace you. And oftentimes you build something and you make it really, really good for specific set of users. and actually the mode is now in your user feedback. The mode is more in whether you listen to them, whether you can rapidly iterate. The mode is in here. I don't think we are yet to, there are so many ideas. There's an abundance of ideas that you can look at regardless. I wouldn't be worried. I feel like, in fact, I do think people in AI fields, I wish they were a little bit more creative and connecting the dots across different fields or something like that to develop really cool new generation and new paradigms of interactions with this AI. I don't think we've cracked this problem at all. A couple of years ago, I was telling some people, I was like, you kind of want to build for the future, so it's like, It doesn't necessarily matter whether the model is good or not good right now, but you can build product ideas such that by the time the models will be really good, it will work really well. And I think it just happened naturally. For example, I don't know if you're aware of the CLIP artifacts. And I feel like early days of Canvas was back in like 2022, like before ChatGPT, like writing IDE was like, I know it's not just. But I feel like Claude 1.3 model itself was like not there to like made like really extreme good, like high quality edits, for example, like coding. Um, and I feel like I see like startups like cursor and it's like doing super well. Like, and that's because they like iterate so fast. They like invent like new ways or like training models. they move really fast, they listen to users like massive distribution is like, yeah, it's kind of cool.

That's really helpful actually. So what I'm hearing is that soft skills essentially are going to be more and more important and powerful. You talked about management, leading people, being creative and coming up with innovative insights, listening. There's a post I wrote that I'll link to where I look, I try to analyze how AI will impact product management. and we're actually very aligned. And my sense was the same thing, that soft skills are going to become more and more important. And the things that are going to be replaced is the hard skills, which is interesting because usually people value the hard skills, like coding, design, writing really well. And it's interesting that AI is actually really good at that because it's taking a bunch of data, synthesizing it, and writing, creating a thing versus all these fuzzy things around of what influences, convinces people to do things and aligning and listening, like you said, creativity. Anything along those lines come up as I say that?

I think it's actually really, really hard to teach the model how to be aesthetic or do really good visual design or how to be extremely creative in the way they write. I think I still think ChatGPT kind of sucks at writing. And that's because it's like, it's like bottom math by this creative reasoning. I think prioritization is one of the most important. I think for a manager, I feel like actually AI research progress is all knocked by management, research management is because you have constrained set of compute and you need to allocate the compute to the research paths that you feel the most convinced about. It was like you need to have a really high conviction in the research path to put the compute and it's more like return on investment kind of situation. It's like, okay, yeah, I'm thinking a lot about how do I, across all my projects, which projects are higher priorities? Prioritization and also on the lower level, which experiments are really important to run right now and which are not and cut through the line. So I see prioritization, communication, management, people's skills, like empathy, understanding people, collaboration. I think Canvas wouldn't be an amazing launch if it wasn't about people. And I think it's a wonderful group of people. I get a chance to work with people like Lee Byron, who's a co-creator of GraphQL, and some of the best Apple designers. So cool to see and how do you create this collaboration between people? It's just something that's still humane, I think.

Let me just follow through a little bit. Because I imagine people listening are like, okay, but once we have AGI or SGI, it'll do all this. There's a world where, why isn't all this done? I think it's easy to just assume all that. I'm curious this idea of creativity and listening. why you think AI isn't good at it other than it's just very hard to train it to do this well. Is there anything there of just like why this is especially difficult for AI and LLMs to get good at?

I think currently it's difficult for many reasons. I think it's still like an actual research area and it's something that I think my team is working on is like, okay, how do we teach the model to be more creative and like, the writing. And actually, I'm thinking this new paradigm of life, the models think more should actually lead to better writing in itself. But when it comes down to idea generation or discriminating of what is a good visual design or not, I feel like it hasn't had learned examples from people. to discriminate it very well. I do think it's because there are not that many people who are actually really, well, it's not accessible to models, so learn from these people, I guess. So I guess that's why it sucks.

Yeah, that makes sense. Basically, there's not enough of you yet, researchers teaching it to do these things slash people that have incredible taste and AI creativity that can teach these things. You could argue this will come, but I'm not, we don't need to keep going down that thread. Let me ask you a specific question in this post I wrote, I made this argument that a lot of people disagreed with that strategy is something that AI tooling will become increasingly great at and take over. There's the sense that that's the thing that people will continue to be much better at and you can't offload AI basically developing your strategy. telling you what to do to win. My case is, isn't strategy to just take all the inputs, all the data you have available, understand the world around you and come up with a plan to win. Feels like AI would be, like an LLM would be incredibly smart at this. What's your take?

I think so too. I think, again, you teach the model all sorts of tools and capabilities and reasoning, right? And it's like when it comes down to, as a for canvas right now, it would be very cool to with the model to just aggregate all the feedback from users, summarize me the top five most painful flows and user experiences, and then the model itself is very capable of thinking of knowing how it's being made, figure out how to create a dataset for itself to train on it. And I don't think we are far away from that kind of like self-improvement models becoming like self-improved via like then like the product development is basically kind of like self-improving like it's kind of like its own like organism or something. Yeah, like again, like strategy is like, it's more like data analysis and like coming up was like, like I think what models are really good at is like, Connecting the dots, I think. It's like, okay, if you have user feedback from this source, but you also have internal dashboards with metrics, and then you have other kind of feedback or inputs, and then it can co-create a plan for you, recommendations even. And I think this is one of the most common use cases for ChatGPT2 is coming up with these sorts of things.

That makes sense. Essentially, a human can only comprehend so much information at once and look at so much data at once to synthesize takeaways. And as you said, these context windows are huge now. Here's all the information. What's the most important thing I should do?

Yeah, same as scientific research. Ideally, the model would be able to suggest ideas, new ideas, iterate on the experiment or given the empirical results of the previous experiments, how do you come up with new ideas or methods?

Yeah. Okay, so just to close the loop on this conversation, this part of the thread is the skills you're suggesting people focus on building and leaning into soft skills like creativity, managing influence, collaboration, looking for patterns, is that generally where your mind is at?

Yeah, I'm thinking a lot about how do we make our relations more effectively? And I think this is mostly management, I guess. It's like, how do you organize research teams or generally teams, combine composed teams such that they will be at the maximally succeed or at the maximum performance of what can possibly If you can literally create the next generation of computers, it's just the matter of conviction and the way you manage through that. It's like scaling organizations or scaling product research, I guess.

Yeah, I think you're basically building this thing and not efficiently doing it is limiting the potential of the human species right now. Mismanagement. within the research team and OpenAI and Anthropic and some of these other models?

Yeah, it's kind of crazy to think about.

Holy moly. Okay, so speaking of Anthropic and OpenAI, you've worked at both. Very few people have worked at both companies and have seen how they operate. I'm curious just what you've noticed about the differences between these two, how they.

Operate, how they think, how they approach stuff. What can you share along those lines? It's more similar than different. Obviously there is a lot of like, there are some differences also comes to nuances. I took culture. I really love Antropoc and I have a lot of friends there. And I also love OpenAI and I still have a lot of friends there. So it's like, it's not about like enemies. I feel like there's like in the Antropoc was all like, yeah, the competitors, there's like enemies. But it's actually like fun, big community and like of people like doing the same thing. I would say what I've learned from Antropoc is this real care and craft towards model behavior, model cost, model training. And I've been thinking a lot about, okay, what makes Claw-claw and what makes ChatGPT? And it comes down to operational processes that kind of lead to the outputs to the model, is the outputted model. And it's like, the reason why Claude has so much more personality and like, is more like a librarian. I don't know, I don't know. I am like visualizing Claude being like a librarian, like a very nerdy or something. It's because I feel like it's a reflection of the creators who like making this model and like a lot of like details around like the character and the personality and like whether the model should follow up on this question or not, like, was the correct ethical behavior for the model to like, in this scenario, it's like a lot of like crafts, and like created like datasets. And this is where I learned that part of like art, I guess, I don't know, but I say like, Anthropic is like much smaller. Like when I joined, it was like, what, like 70 people? When I left, it was 700 people. And like, obviously the culture changed so much. I really enjoyed being like early days startup vibes. and people knew each other as a family, but the culture shifted. I would say I learned from Andarbik that they're much better at focusing and prioritization, very hardcore prioritization, I guess, and they need to do it. But I think OpenAI is much more innovative and much more risk-takers in terms of product or research, actually. you know, like, I don't know, you can, like, your full-time job can be just, like, teaching the model how to be, like, creative writers. And it's like, there's some luxury in this, like, research Freedom that. That comes to scale, maybe. I don't know. But it gives you. It's like, you'll have. I feel like I have much more creative, like, product Freedom to do almost anything, I guess, within, like, open AI, like, I've lost chatbot being to, like. dividend that you want. It's like more like, yeah, probably bottoms up, I guess.

Yeah, that's how I was thinking about it. It feels like opening eyes more bottoms up, distributed, people bubble up ideas, try stuff. There's more and that leads to more products launching. I imagine more things just kind of being tried versus more of a, let's just make sure everything we do is awesome, great and craft and thinking deeply about every investment. That's really interesting. I've never heard it described this way. Karina, we've covered so much ground. This is going to help a lot of people with so many ways of thinking about where the future is going. Before we get to our very exciting lightning round, I'm curious if there's anything else that you think might be helpful to share or get into.

One of my regrets, I guess, when I was early days at Antwerp was that, I think there was some luxury of the time to pre-ChiPT to actually come in with a bunch of ideas and prototype almost every day. And I think we did a lot of cool ideas. Claude and Slack was actually one of the first tool you use products. Claude could operate in your workplace now. It's kind of cool. You add Claude to summarize the thread. So maybe you have an entire conversation with someone and then you want a summary of what happened. You can say, add Claude to summarize this. Also, it was really fun to even iterate on the model itself. It's like when you just talk to the model in Slack forever. It created some social element. It's kind of cool. It's kind of like me journeying in this Discord. People learned so much about prompting and how to work with Claude. I feel one of the features that was early tasks prototype was every Monday Claude would just summarize the entire channel. or like every Friday we just summarize a bunch of channels and give the news about the organization or something. So it's kind of a really cool form factor. I think thinking about form factor is a really important question in AI, especially we haven't even figured out how do we create an awesome product experience was like, oh, serious models. It's like the paradigm between synchronous, real-time, give and answer paradigm into more asynchronous paradigm of agents working on the background. But then now the question is, the agents should build trust with you, right? And trust builds over time, which is like with humans. And you start this collaboration, which is why I like a collaborative, this collaboration model was like you and the model is so important because you both trust and the model learns from your preferences so that it can become more personalized and it will start predicting the next action that you want to take on the computer or something. And it's kind of more predictive, much more... We went from personal computer to personal model, basically here.

Why is it not a thing? That seems like such an obvious feature that every Llm should have is a Slack bot version of them. Is that a thing I can have you install or is that not a thing right now?

I know that Claude and Slack was sunsetted in 2023 or something, but that's because I think it was after Chai GPT it was mostly the focus on consumer use cases or enterprise use cases. I think the form factor of Claude and Slack is like, was kind of constrained a little bit when you wanted to develop new features.

I want that.

I know that ChatGPT had like Slack part too, so I don't know, like maybe it will come back.

All right, I would pay for that. Any other memories from that time of early days? Because that's a really special place to have been as early days Anthropic. Any other memories or stories from that time that might be interesting to share?

I think the very first launch when we felt like, When Clickstream used again, it was like a 100k context launch. It's like when the models could input the entire book and give you a summary of the book or something, or the entire financial or have multi files, financial reports, and then give you an answer to the question, to very specific question. I think there was something in there that kind of like, Oh my God, this is like a really cool new capability, not like model capability, but more like the capabilities that came from the product form factor itself rather than like the model capability as much. I think like other prototypes that we were thinking about, like, yeah, like there's like one part of a clockwork spaces and it's like, kind of the same idea of Claude and I would have this shared workspace and that shared workspace is like a document and we can like, do it in the document. And I feel like sometimes the ideas, like, probably ideas lag and they lag for like two years, just like in this case.

It's interesting there's these milestones that kind of open up our view of what is happening and where things are going. ChatGPT, I think, was the first of just like, wow, this is much better than I would have thought. you talked about 100K context windows where you could upload a book and ask you questions, have it summarized. I actually use that all the time when I have interview guests and they wrote a book. I sometimes don't have time to read the whole book, so I use it to help me understand what the most interesting parts are. And then I actually dive into the book just to be clear. And then, I don't know, maybe like voice was another one where you could talk to say chat GPT. Are there any other moments there that you're like, wow, this is much better than I thought it what is it going to be?

Yeah, I think like the computer use agents, like the model operating your desktop. And you can essentially think of like, you know, new kind of like experience where the model can learn the way you browse. And from that preference, it can just like browse as just like you. And it's kind of like simulation simulated like persona. And it's actually very similar to the idea of like, okay, like maybe Sam Altman doesn't have a lot of like time. Maybe I want to like talk to like his simulated, like his simulation and ask like, oh like for example, like yeah, like I really appreciate some of the technical mentorship from like Jakob, but he doesn't have a lot of time. So it's like, I really want to like ask him these questions, like how do you Simulated environments like this would be really cool. That's a great place to plug LennyBot.

I have one of those. It's trained on all of my podcast and newsletters. And it sits on many models. I don't know which one exactly they use, but it's exactly that. And it's not even me, it's all the guests that have been on the podcast, all the newsletters I wrote. And you could just ask it, how do I grow my product? How do I develop a strategy?

And it's actually shockingly good. Do you feel like it reflects who you are? Yeah, yeah.

The best part of it is you can talk to it. It's built, there's an 11 Labs voice version that's trained on my voice on the Fitness Podcast, and it's actually very good. And people like have told me they sit there for hours talking to it.

Wow.

And somebody told it, interview me like I am on Lenny's podcast, ask me questions about my career. And he did a half hour podcast episode with Lenny Buchwalter.

Oh my god. That's so fun.

It's incredible. Future is wild.

Yeah, I think content transformation is like, you know, like I would imagine sometime, like, you know, when you generate a sci-fi story in Canvas, like you can like transform this into like audio blog, like very natural, like content transformation from one media to another media. I think like one of my earliest inspiration is one of the last episodes of Westworld, where I want to spoil, but where Dolores comes to her work at that time and she comes to this new workspace and she starts writing a story. And then as she writes a story, a 3D virtual reality starts creating on the fly. So I kind of want to I hate that. Kind of cool.

Wow. Speaking of medium, I guess I was wondering if I should go in this direction or not, but real quick, Kevin Weil slash Kevin Wheel, I don't know exactly how to pronounce his last name, the CPO of OpenAI. Is it Weil or Wheel?

I think Wheel.

Wheel. Okay. Okay. Let's just say that. Okay, Wheel. He did a panel at the Millennium Friends Summit last year, and he made this really fascinating point that chat is a really interesting interface for these tools because they're just getting smarter and smarter and smarter and smarter and smarter. And chat continues to work as a paradigm to just interact with them similar to human. You could talk to Albert Einstein, you could talk to someone not very smart, and it's all conversation still. And so it's a really flexible way to interact with increasingly Good intelligence. At some point it'll not be so great. And you were talking about all these ways that you're adding additional ways to interact. But it's interesting chat proved to be a really powerful layer on top of all the stuff.

Yeah, that's real cool. I feel like chat also has like social element, which is like very humane. It's like, you know, you sometimes want to like get into group chat and like, you know, having conversations with AI is kind of like a group chat in itself. It's like massive thing. I see this idea of like, How do you build features like this? I see tasks as this general kind of feature that will scale very nicely as the models will develop new capabilities themselves. The models will be able to do better searches and come up with more creative writing or render React apps and HTML. like apps and you can have every day a new puzzle for you, like every day continue the story from the previous days. It scales very nicely.

You mentioned something as we were getting into this extra section that we ended up going down is this idea of the agents using a computer. I know this is actually something you are going to launch today, the day we're recording it, which will be out by the time this comes out. Called Operator. Can you talk about this very cool feature that people will have access to.

Yeah, so I unfortunately did not work on that, but I'm really, really excited about this launch. It's basically a machine that can complete the task in its own virtual computer, in its own virtual environment. You can do any literally task, like order me a book on Amazon. And then ideally the model will either follow up with you, like which book do you want? Or know you so well that it will start recommending, oh, here's the five books that I might recommend to you to buy. And then you hit, yeah, help me buy. And then the model goes off into its own virtual little browser and complete the task and buy the book on the Amazon. And then if you give the model credentials, credit cards, obviously it comes with a lot of trust and safety, then it will just complete the thing for you. It's a virtual assistant.

It's interesting how this just sounds like obviously this should happen. Why is this not yet a thing? Which is also mind blowing that we're just assuming this should exist, just some AI doing things for you on a computer. you just ask it to do. Like, it's absurd.

It's actually really hard. And I think, like, you're still cracking this, but I feel like, I don't know if you use, like, topple. It's like a pair programming product. No, but I don't know if you love pair programming. So if you.

Oh, yeah. Shopify uses this. I remember it came up on a podcast episode.

Oh, nice.

Yeah.

So it's. It's a very cool product where you can just, like, call anyone. at any time and then like share screen and the other person can have access to the screen and start literally operating your computer. And it's very real time, the allegiance is very high quality. And it's just like, I kind of want the same. I want to pair program with my model and the model should even talk to me. draw very specific section in my code and VS code and tell me, I will teach me and we can have different modes. It's like right here, it is like a product right here. I don't know. Some people should build that.

Sounds like a startup just got birthed from someone listening to this. You mentioned that it's very hard to do this agent controlling a computer as you and helping out. What makes it so hard? for whatever, however much you can explain briefly.

Much of it is like, because right now the models operating on pixels instead of language or whatnot, pixels is actually really, really hard for the models because of perception or visual perception. I think there's still a lot of multimodal research that's going on, but I think language scaled so much easier compared to multimodal because of that. Another thing that my team is working on is how do you derive human intent very correctly? Sometimes, does the model know enough information to ask a follow-up question or to complete the task? You don't want an agent to go off for 10 minutes and then come back with an answer that you didn't even want. That actually creates much more reverse user experience. And this comes as teaching the model people skills. It's like, you know, what do people like? Kind of like creating the mental model of the user and care about the user in order to ask certain questions. Actually, that part is hard to the models.

That relates to what we talked about earlier, kind of the soft skill people skills pieces. not where these models are strong yet. Okay, I'm going to skip the lightning round. I want to ask just one question from the lightning round. Something funny. Okay, so when AI replaces your job, Karina, I'm curious what you're, and it gives you a stipend, gives you a monthly stipend. Here's your, here's your salary for the month. What would you want to do? What do you want to spend your time on? What will you be doing in this future world?

I've been thinking about this. A lot of times I have, I feel like I have a lot of jobs options. I would love to be a writer, I think. I think that would be super cool. You should write short stories, sci-fi stories, novels. I really like art history. You know, as a conservationist in the museums, we just try to preserve art paintings. But just like painting through a lot of them. I think that'd be really cool to do.

Yeah. That sounds beautiful.

I don't know.

What I'm hearing is you need to nerf these models to not get very good at writing so that you can continue. Although at that point, you don't need to do it for them. Like, you don't need people to buy it. You're just doing it for fun. So it doesn't even matter if they're incredibly good at writing or art, art conservation. Oh man, what an episode or a conversation, what a wild time we're living in. Karina, thank you so much for being here. Two final questions. Where can folks find you online if they want to reach out and follow up on anything? And how can listeners be useful to you?

You can find me, I'm on Twitter, it's @KarinaMihai. You can also shoot me an email on my website. And my team is hiring, and so I'm looking for research engineers, research scientists, as well as machine learning engineers, people who come from product engineers who want to learn model training. I'm actually hiring for my team. My team is called Frontier Product Research. And we train models, we develop new methods, but for product oriented outcomes.

What a place to work. Holy moly. What's the best way for people to apply for these very lucrative roles?

I think you can shoot me a DM on Twitter or I'm yet to create a job description. Okay, this is the job description.

Or you can apply under like post training team. Okay, so you're going to get a flood of DMs. I hope you're prepared. Karina, thank you so much for being here.

This was incredible.

Thank you so much, Lenny.

Bye everyone. It was fun.

Thank you so much for listening. If you found this valuable, You can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lenniespodcast.com See you in the next episode.