AI Avatars & the Future of Video, with HeyGen CEO Joshua Xu and Benchmark's Victor Lazarte

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg.

Hello and welcome back to The Cognitive Revolution. I'm Nathan's AI avatar, created on HeyGen, and I'm here to introduce Nathan's conversation with Joshua Shih, founder and CEO of HeyGen, a leader in AI-powered video creation, and Victor Lazarte, general partner at Benchmark, which recently invested $60 million in the company. Over the last 2 years, while generative AI for images and text were crossing critical thresholds for usefulness and hitting mainstream adoption, AI-generated video has remained, for the most part, a future promise, something that AI watchers could clearly see coming and and which has provoked much speculative analysis, particularly about the future of the entertainment industry and the risks associated with convincing deepfakes, but still, for practical purposes, a novelty technology that produces uncanny valley outputs. That has started to change recently with OpenAI showing off Sora's general world modeling capabilities, Google previewing Veo, China's Kuaishou Technologies debuting their Kling model, and most recently Runway's Gen-3, among many other options starting to hit the market. But even today, these products remain quite unwieldy and inconsistent. And, as a result, impractical for most people to actually use in their day-to-day work. Against that backdrop, HeyGen's success stands out all the more. By focusing their product development work on practical business needs, emphasizing the value drivers of quality, consistency, and control, while taking a best-tool-available approach to technology, ensembling their own AI avatar models with best-in-class tools for scriptwriting and text-to-speech, they've established a market-leading position in the generative AI for video space and are now serving more than 40,000 businesses, And earning $35 million in annual revenue. In this conversation, we discuss what customers are using HeyGen for today, which includes creating new video content that wouldn't otherwise have been possible, translating video into more than 100 local languages, and personalizing video down to the individual viewer level. And Joshua and Victor paint a picture of a world where video content is not just more easily produced, but fundamentally transformed. Imagine immersive video experiences that adapt in real time to viewer interactions, or AI avatars that can engage in live, dynamic conversations. These aren't just disruptive alternative production methods that allow us to make the same content with fewer resources. They are potential paradigm shifts in how we think about video in general. As someone who's spent much of the last 3 years applying AI to video creation at Waymark, with business results that, while not as explosive as HeyGen's, are objectively strong in their own right, I can say that such reminders to zoom out and challenge oneself to think bigger and stranger are always worthwhile. As it happens, we had a recording issue in this episode that turned out to be the perfect opportunity to demonstrate the power of HeyGen's product. For whatever reason, Joshua's video track, which looked fine while recording, was corrupted. We considered releasing this episode in audio-only form, but ultimately decided to recreate Joshua's portion of the video with HeyGen. After this intro, the video you'll see of Nathan and Victor will be real, and the video you'll see of Joshua will be created with his HeyGen avatar based on the transcript of the original conversation. I have to say, it came out amazingly well. Toward the end of the episode, we also discuss HeyGen's industry-leading approach to trust and safety. At a moment when most AI application developers are just trying to make their products work, and many are ignoring risks of misuse and abuse entirely, HeyGen has done an outstanding job of taking the responsibility that comes with the power of their product seriously, and has implemented robust consent mechanisms to protect famous people from being impersonated and other safeguards designed to protect the public from political misinformation and scams. Their thoughtful approach demonstrates that rapid innovation and ethical considerations can go hand in hand, and I recommend that any entrepreneur or AI product builder looking for inspiration in this area, check out HeyGen's product experience or read my Red Teaming in Public thread that breaks down their defense-in-depth strategy. And by the way, if anyone listening would be interested in contributing to the red teaming or other safety testing of public and soon-to-launch products, please send me a DM. The Red Teaming in Public project has been a bit quiet lately as another team member has taken the lead and is building some foundational infrastructure behind the scenes, but we are still very much interested in connecting with folks who enjoy testing and breaking AI product guardrails. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to review us on Apple Podcasts or Spotify, or just share online with your friends. Of course, we always welcome your feedback and your AI advisor and AI engineer resumes via our website cognitiverevolution.ai, and you can always DM me on your favorite social network. For now, I hope you enjoy this glimpse into the transformative potential of AI-powered video with Victor Lazarte of Benchmark and Joshua Xu of HeyGen.

Joshua Xu, founder and CEO at HeyGen, and Victor Lazarte, general partner at Benchmark. Welcome to The Cognitive Revolution.

Thank you. It's great to be here.

Thank you for having me.

My pleasure. This is going to be fun. So for folks that don't know HeyGen, and I suspect most probably are at least passingly familiar, you guys do generative video, a space near and dear to my heart, though I come at it from a pretty different angle product-wise. And I thought it'd be interesting just to kind of start off with, like, briefly, how did you get into this business? I heard this story on the No Priors podcast, but give us kind of the brief background as to how you got into HeyGen. We can talk about where the business is today. I really am interested in then going into more deeply into like use cases, technology, and especially the trust and safety stuff, given the nature of the product that you've built. But the floor is yours for the backstory.

Yeah, sure. Yeah. So I, before founding HeyGen, I was working at Snap starting from 2014 to late 2020. So initially I was working at advertising team, basically building out a lot of our ad system at Snap, working with a lot of advertisers, helping them to get the right ROI on the Snapchat ads platform. And then later on, I switched my team to work on camera, AI camera. Basically, you know, I still remember back then in 2018, technology, there's nothing called generative AI, but there's some technology called GAN. Where you can actually generate something that does not exist in the world. And during that time, you know, that was just the first time I saw a computer can generate something that does not exist in the world and still feels high quality and highly realistic. And I was staying very frontline about those technology and just had a feeling that that could potentially change the way how people create content. So, you know, especially if you look at the past, I would say almost 10 to 15 years, the new content platform evolved with the raising of a mobile camera. And since iPhone 4 came out, this application such as Instagram, Snap, TikTok, like unlocked so many creators to be able to create good contents. But, and especially working on the camera software, we still see many people are not able to create good content with the mobile camera. Either they are shy in front of camera, They don't have time for the camera, or they're not good at performing in front of a camera. And we felt that if we have a way to really replace that component, replacing the camera, we could have a way to really unlock the visual storytelling for everybody. And that's how we kind of like found HeyGen. We wanted to replace camera because we think AI can create the content. AI could become the new camera. And we want to use AI to generate a video instead of having people to actually film it with a camera. That's how we get started founding HeyGen back in December of 2020.

Cool.

Well, it's been— your timing is certainly impeccable from the original adversarial network generative AI technology. Obviously now we've got a much more powerful set of tools to choose from, and you've grown quite a business on this technology with, I understand, 40,000+ businesses that use it. $35 million in annual revenue, which is pretty impressive. And obviously a big raise from Benchmark to put a further feather in your cap as well. So a lot of great traction. I'd love to dig in a little bit on like, what do people use this product and platform for? I feel like people had told me for a long time, oh, you should do a podcast. I'm always told I have a voice for radio. And sometimes they also say I have a face for radio. But certainly the voice, they say. And I was always like, oh, man, I don't know. The world has so much content in it, and I'm not that charismatic. I'm not that funny. It was only when I got totally obsessed with AI to the point where I felt like, well, actually, in this narrow domain, I might have something that would be interesting enough that people might want to listen to it. So I guess for starters, I'm like, In a world that's so awash with content, like, where do we need more content? What kind of content do you see not existing that you're allowing to come to exist? And why does that matter?

So I guess first of all, I would start it with the audience. The consumer lives in a video-first world. And everybody wants to watch videos. There's more than a billion hours of videos being watched on YouTube every day. And the example I used was that, you know, I had some issue with my car. the other day and my Tesla. And I didn't actually look at the manual, but I actually looked up a video on YouTube because like video is just much easier for me to get the information I want and for me to consume as a consumer. I would say one thing we learned from customer is that every business wants to make more videos. But the problem is that video is just expensive to make. What we are trying to solve the problem here is that helping the business to be able to make more videos, to unlock their needs, to match up to pace with what their customer needs. So that's one angle to look at it. And I think Nathan, you asked a little bit about how people are using it today. And I would generally just like categorize into 3: create, localize, and personalize at the current platform of HeyGen. What I meant by create is that people can come into the platform. They can either create their own avatar or select the stock avatar we have. And then pick up a template and type the script to generate a video. That's the great piece about that. This is great for product explainer, product demo, product announcement, and some of the training, learning development videos. And the second piece comes to the localization, where we can take your existing videos and translate that into more than 40 different languages. We have a feature called video translation where we can take the original video, preserve the voice of an original speaker and also the facial expression, but in just different languages. And, you know, lots of companies use it for localizing their content, especially for some large corporates where they have a specialized localization team. We help them really, really shrink down the workflow cost and time by more than 10x. The last piece, I would say, just personalization, where you can actually highly personalize your message. We have seen personalization at email, right? Basically, you've got email who say, hey Nathan, do you need this service or that? But we're inventing a way that you can do that in video as well. I think personalization on name is just one starting point, but we can also personalize the content itself. What actually works for you in your business, combining with the power of LLM, like companies like Publicis Groupe create more than 100,000 Personalize the email as a video format and send it to internal employee as a thank you video. And those are the main 3 categories I would say in the use case of agent.

So create, localize, and which is translate and personalize. I'll start in the middle. The localization one to me seems like an absolute no-brainer. I think one of the things that's kind of most exciting about the AI moment is just how many barriers it stands to break down and how much better we'll be able to communicate with people around the world. I'm going to Brazil next week, and I don't have to worry really at all about translation anymore. I can just have my phone do it. And that's going to be such a freeing experience relative to certainly whatever I would do in the past, right? Get out my phrasebook or whatever. So it makes a ton of sense to me that businesses would be like, sure, you're telling me I can affordably translate my content into however many languages and reach people all over the world? That seems like something that Is probably a pretty easy sell. What does the cost look like on that? Just to kind of make the ROI case for a business. If I'm in ecommerce or whatever and I've got, let's say, 100 product videos and I want to translate that into 100 languages, I'm going to make 10,000 videos. What sort of— I can only imagine what my cost would be if I was going to do that old school. The obvious outcome would be just to not do it. In the HeyGen world, what would that cost me? Like, give me a little sense of the practicalities of doing a project like that.

Yeah, sure. So I think really the cost about using AI to generate that, to do that, is really the cost of a GPU in that case, the GPU compute cost. Typically when we see the doubling rule, so I would put it into 2 ways. But traditionally, when we talk about localization, we mainly talk about hiring a voiceover actor to dub it and then put it back on the original video, right? That costs about— it really depends on which part of the region we hire the voice actor, but let's say somewhere around $10 to $20 per minute. But the problem is that with that approach, your video still doesn't have the engagement or facial expression that match with your voice, right? I think what our approach, our method on video translation is that we actually made that engaging with a new language, new voice. So I wouldn't see the value purely on reducing the cost. Yeah, we reduce the cost by more than 10x and making the time much, much, you know, probably 100x faster in that case, because it usually takes like days or weeks to really coordinate the voice actor, but now you can do it one click with AI. I think the other important value prop here is that people always think that, you know, a video dubbed in different languages is always like the second version, right? The version that's less engaging because I kind of want to look at the original video, but just look at the subtitle. But now you can really enjoy the native language version, pretty native way, with the speaker actually with the lip sync movement matching the new voices. I think there's another value of that where our version of the video translated video is actually much more engaging than the traditional dubbing videos.

That seems like a killer to me. I don't know if you are inclined to say, but is that like a big part of the business?

I mean, I think that those 3 are the 3 main pillars, like create, localize, and personalization. I couldn't share the specific details on that, but that's definitely one of the most powerful and popular use cases among our customers.

Hey, we'll continue our interview in a moment after a word from our sponsors. Cool. On the create side, I wonder how you think about I mean, obviously video is a vast world, right, with an infinite range of things that can be created. I wonder what sort of taxonomy you have for that in terms of, do you organise it by the kinds of content that are being created or the sorts of people that are doing creation? How much do you see being created that would have been created previously, but with more time and money required versus how much stuff is stuff that just couldn't have been created or wouldn't have been created before. And do you guys mostly create inputs for people to take to other projects, or do people actually leave the HeyGen product with a completed video? I could imagine somebody might generate something with an avatar and then go use that in Adobe Suite as part of a bigger process. But I know you also do have the, the like studio portion of the product. So yeah, just tell me everything about who's creating what.

Yeah. So who's creating, what are they creating? And maybe the last one is like, how are they using it after they create it on HeyGen? So HeyGen is a very horizontal platform. There are many, many different use cases ranging from marketing, sales, learning development. And, you know, like we mentioned, localization. And I think another aspect is also that there's a variety different of segment of customers on HeyGen, like ranging from some consumer customer, small business, midsize, and some of the largest global, you know, Fortune 500 company as well. We see us as creative tools and where we want to build a tool for everybody that they want to create videos to come pay Jen to create videos. I would say in terms of content that actually being created, I think you asked a good point. Is it more about unlocking their existing workflow, make that faster, make that cheaper, or is it more about unlocking the audience by making video that they couldn't be doing that before? I think we talked about a little bit on the video translation example. I don't think translating a video, it is a term people thought about that previously. We, you know, really opened up that new market and making it possible for people to localize their video. And that's a very similar thing applied on the avatar video side. The way how I put it is like this, you know, assuming a business needs to make Let's say 100 videos a month, right? Just for example. And maybe 10 of them, they could actually come to HeyGen to create with some sort of like spokesperson, like human presenter videos. And we made that 10 videos, even though that's only 10% of the total video they're making, but we made the 10 videos like much, much cheaper and faster. And as an outcome of that, Instead of not making 10 videos, instead, because a better tool out there, they're making 100 videos out of that original 10-video demand. And that's one way to look at it. I think another way is that I'll give you one example would be, let's say a small business customer. They usually don't have a lot of budget on working with a video agency to create videos, but they really want to leverage video as a format to do go-to-market for external marketing. And now HeyGen, it is a perfect tool for them to really unlock that. And in that case, we can imagine that we are really unlocking the video they couldn't be creating before. So I guess it comes back to the point, HeyGen is a creative tool. We help to shrink down the creation costs out there, and it just has a value and effect on both sides.

Yeah.

The last piece is about how people are using it. Either they create the end outcome from HeyGen or they put it onto other video editors. The way how I look at it is that, so let's say we largely categorize the customer, the audience into 2 groups. One group is that I call the professional video creators where they have access to Adobe Final Cut Pro and other Categories that people actually don't know how to make a video before is they don't know how to use those sophisticated softwares, right? You know, the first piece of the customer, I'll call that, let's say, 1% of the total business users. And they would probably use HeyGen as a generator footage. And the value here is actually HeyGen helped them to replace the camera, the process. Where they used to have to use a professional camera crew to film it with the footage. Now they don't need to do it anymore. They can use it on HeyGen and plug the footage into Adobe or whatever editing tool they're using. That's for that 1% professional video creation players. But actually, for the rest of the 99% users, they couldn't actually make a video before. In this case, they would just finish everything in Not only the camera part, but also the editing piece on HeyGen, producing content directly that way. Yeah.

So do you see people creating stuff? I mean, obviously the— and I'm thinking actually for the introduction to this episode, I might go clone myself and like have my avatar read the intro essay instead of doing it myself. You know, the company is most well-known certainly for the avatars. Which, and when you go into the product, which I've done, I always get deep on the products before the interviews. I noticed that there's like a handful of pre-done avatars that I can choose from, or I can clone my own. How often do people, I mean, that seems to be the thing that people are like initially hearing about that they come to the product for, I guess. How often are they using like an off-the-shelf avatar, a cloned avatar, or no avatar? I imagine in some cases people might be making videos that doesn't even feature an avatar, but there's so much emphasis on like the deepfake technology of the clone. But I wonder if that is actually what is driving most of the value or if it's just kind of convenience of video generally that's driving the value.

Yeah. Yeah. First of all, I would say the voice-only use cases where they don't have an exposed person generated, that is not the major use case of HeyGen, even though probably you can still get around that to generate video like that. The way we really look at it, whether that's a stock avatar or your own avatar, it's really about how we design the user journey. And I think the stock avatar, the major benefit out there is to help users to get the value of this technology and can quickly experience what's possible with that. I think the user journey is always trying to help customers create their own. digital twin so that they can use that on their own businesses. So it's not about the end use case. It's actually about the user journey.

So I take that to be most of the people are actually doing the clone.

Is that right? Most of, I mean, I would put it this way. Usually if you look at a customer journey, they would usually come to HeyGen, test it out with the stock avatar and trying to see the value or even test out how effective their content In terms of performance. And then as a next step, people will come to HeyGen to create their own avatar and continue to strengthen that, build brand-specific identity around it. And that's kind of a user journey to drive that. Gotcha. Okay, cool.

Still staying on the avatars for a second, where do you see these working well versus not working well?

And how do you think that's going to change over the short term?

I heard on the No Priors podcast you went into some detail on, obviously, when you're making marketing videos and advertising videos in particular, the ROI on that is super determined by how good the content is, how engaging it is, whether people swipe immediately or not, right? Versus something where if it's like a required corporate education experience, you have to finish it kind of regardless of how engaging the presenter might be. So I wonder how you see, where are we right now on the spectrum? I'm definitely someone who expects that wherever we are right now, we probably will get to things that are outside the uncanny valley. But today, are we still— how do you think about the uncanny valley? Are we in it? Are we leaving it? Does it depend on the use case, like education versus top of funnel advertising? And where is the technology? good enough to win versus maybe not quite there yet? And again, like, how's that going to evolve?

Yeah. So I think you mentioned something about the external use cases and internal use cases. The quality bar there are different. And especially for external marketing advertising, the quality there would be much, much higher. And because, you know, brand or business is going to put a big budget on that. to 5 ads. I would look at it this way. So assuming we try to plot the quality, you know, as a spectrum like this, right? And I think certainly we have surpassed in a lot of aspects in terms of the uncanny valley already. Like, for example, if you look at some of my avatars, you probably couldn't tell whether that is avatar or a real human presentation out there. But I think the tricky part is that Let's come back to a question. Why would people love watching videos instead of actually reading text or just like listening to a voice clip, right? Because I think one important piece of a video is that it's just more engaging for the audience. And I think the question comes from not like what makes a video look real, it's actually how to make a video engaging. And that piece is still something I think needs a lot of development and improvement out there. I'll give you one example is that Nathan, you're probably really good at performing in front of a camera. You do podcasts, like interview, like pretty much every week, right? And what I mean in terms of like replacing that for you, having a realistic generation for you is not enough. I think there's something else out there. I was talking to The other person about some feedback on the product. And she mentioned something about conversation flow, right? Okay. What does conversation flow mean in the video generation problem? And that's actually very subtle, but it's very, very important to make a video engaging. And the other one is that the conversation flow usually comes with a natural movement of a gesture. Like I'm just using my hand to help me to articulate some problem. And I think that's another challenging piece of that. And then the last piece is that, like, generally more emotion expression. And because you can say the same word in different tone and different emotion, and actually it represents different meaning behind it. I think those areas still is an area we're actively working on to improve. And that is actually breaking on to where we can surface more and serve more like high-end, high-quality external advertising or marketing use cases.

I think your comment about the target not necessarily being full realism, but just being high engagement content is a pretty profound one. It's all happened so fast over the last couple of years, but I remember maybe, I don't know if it was a year ago or 18 months ago or something, but I remember saying, as we were looking at some of these early Like, image generation was already getting pretty good at this point, and video was definitely not as far along as it is today. And I recall having this hypothesis that maybe we'll sort of adjust as well. Of course, people are going to try to make the video generators more lifelike in some respects, but also there's sort of like a super stimulus feeling to some of the early stuff where it has this almost psychedelic sort of wavy, form that it seems like in a way it also kind of recalls some of the interpretability results where you look at what sort of visual input makes a certain neuron in a vision model light up. And it's like, wow, it's sort of a trippy thing that maximises the response of this one particular neuron. And I feel like people are kind of racing toward realism and at times forgetting that in fact, cartoons are really popular and there's Lots of other form factors besides like purely lifelike video that maybe can win. And this is maybe a good question for you, Victor, too, because I know you have a background in video games. Like, where do you guys think the future is of content? Is it about realism or is that kind of a lazy person's way to think about like what really matters?

I personally think that Like realism is not the most important thing. And like to Josh's point, it's all about the engagement, right? And I think given the choice of consuming content in different formats, I think it's pretty clear that humans choose video. And one way of like having it be more engaging is realism, but in entertainment, there's a lot more than just realism, right? I think one of the things that first got me super interested in HeyGen is I started and ran a mobile gaming company for a long time. And in mobile gaming, user acquisition was always a big part of the business, right? And in the beginning, people would show static ads and eventually it evolved to video ads. And video ads is a huge part of the mobile gaming industry. So for example, for my company, we had 100 million video views per month. So just getting the video to convert at a higher rate is just so fundamental to the business. That's one of the largest KPIs, right? And I think that translates to other businesses as well. So companies, they want to communicate with stakeholders both internally, but more and more as the quality of videos get better, more and more external stakeholders like, hey, how do you make these videos more engaging? And I think In my experience, what really worked was not necessarily like, oh, how do we make this higher fidelity? But it's more like, how do we test a bunch of different things? And the key to testing is how do you get the price to be lower and how do you get the iteration speed to be faster? So I think that's a very powerful thing that HeyGen does for businesses. You're able to create videos Faster at a lower cost, which means you can try a lot more stuff.

Do you see an application in gaming? Because I'm also kind of thinking about the personalization side.

Yeah.

So I think what got me excited with HeyGen is like what we're seeing now, I think it's like we're scratching the surface. And I think the thing that impressed me about the company is like just the product velocity. And so when you look at some of the new products, for example, like streaming avatar, Which is your avatar. So it's a digital clone of a person that talks to you in real time, right? So instead of like, hey, I want to create a video that someone's going to watch asynchronously, it's like, no, it's like someone's going to talk in real time with a bot and like HeyGen powers that technology. And like no one has that technology. And I think it is one of the most exciting things about the company because this new wave of AI started with like ChatGPT capturing people's attention. And I think what captures people's imagination is like, okay, people are now talking with a bot. But the first iteration of ChatGPT is like you're talking with a bot, but you're typing, which is suboptimal. And with the GPT-4o announcement, what was really cool is like, hey, now like talking with voice, it's kind of cool. And the next step of that It's not like a voice call, but it's a video call and streaming avatar. We have the best product in the world to do that. And tying back to your question around entertainment, I think there's just so much more pleasurable for people to engage in this way. So we don't yet know, and I'm happy that we're a platform that provides the technology, but we don't yet know what companies will do with that technology. But if I had to guess, there'll be huge applications in entertainment. It's like, hey, I wanna talk to virtual friends. I wanna talk to a pet. You know, like we have these avatars that some of them are cartoons, like some of them like have these different features. So I guess it's a long way of saying that we're just getting started.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, I think that those, even the non-human avatars are really an interesting concept. When I think about the mass personalization along the lines of email, there I'm like, oh man, I don't know. I guess a couple things. One is like, If I ever get to the point where I'm getting an AI-generated video from my boss, I'm going to feel maybe a little bummed out about it. Just like, wait a second, you didn't have time to actually talk to me? Or something feels weird about that. I also find in what little exposure I've had to these sort of large agency, large brand personalization projects in the past that they love the idea, but then they're also— and I wonder if you've found this too, but they've typically been very shy about actually meaningful personalization. They love to get that name in there, but then I've seen solutions in the past where it's like, we'll do the top 100 most frequent names, then after that we'll just say, hi friend, or whatever. And they're like, yeah, that's great. We love that. But then are we actually going to do any other customization deeper? Well, we'll maybe put how much you spent on your utility bill this month or something, but it's very local, very sort of Programmatic and it's like almost, yeah, do you really even need AI for some of these things? But then I imagine this other world where it's like you're on an adventure in a dynamic world. Maybe the whole world is getting generated for you. And now it's a lot easier for me to see how that real time and the personalization to whatever adventure you're on, experience you're having, makes a lot more sense.

Yeah.

Do you think I'm analyzing this in the right way, or do you see big brands as kind of actually being more adventurous than I give them credit for?

Yeah, I mean, I think one way I would look at it is that, first of all, I think by looking at personalization combined with the generation only at the name variable level, it is still a pretty narrow view, in my opinion. I think the bigger implication behind that is actually, so first of all, video can be generated on the fly. Second, video can be highly personalized. People started with name because that's much easier to get started. And that's an immediate use case for that. But which also means that you can personalize the content itself. What is the messaging hitting that customer? What is the story behind that video?

Right.

And I think we have been always get used to living in a role in the past 20 years when you watch a video. you and I have to be watching at the same one, but it doesn't need to be like that, right? So we could be totally watching 2 different angles to talk about the same story, the same thing, the same topic behind it, and the video could be presented in different ways. One example I want to use, like the, the example Victor brought up, is that, hey, let's say you have a video getting 100 million views every month. You want to optimize that. I think the conversation will always come into, okay, I think most people would probably like this, so let's change the video that way. And then you keep iterating. And then every time you are optimizing that, you sacrifice some sort of customers. You're basically optimizing for the rest of the broader audience to reach the global maximum. But still, there's only one video out there. But imagine a world if we break all this down we can actually serve every single customer landing on this page a different version of that. And we have AI behind that to support it that way. I think we can achieve the global maximum in terms of ROI and optimization there. And I think when we really look at it, how is that possible? Like highly personalization is one technology behind it. And second, I think what's really important is also about Can you actually generate the video on the fly? And when I give you a variable and then you generate the story behind it, like one example would be the streaming avatar, but streaming avatar is really the starting point of that. But imagine a world that we can actually stream the entire video, not only streaming the watching experience, but also streaming the generation experience. I think that is just something we will look at it and feel very excited about the future about. generating the video.

Yeah, and just adding on this, I think the technology to produce these personalized videos is so new, and some of the applications that people make, like, we have to test a bunch of things, and in the beginning, not everything resonates, right? So for example, like, you can receive a video from your boss that is just your name being personalized, like, and that doesn't make you feel good, but at the same time, A video that is personalized, not to just like, oh, I'm gonna make you feel like this was made exclusive for you. That's the only value. No, imagine it's a video where like we intake all the information about the stuff that you care about, and we personalize not to make you feel special, but we personalize because we actually gonna deliver the contents that you care most about. And I think as we find more and more of those applications. Then people start feeling like this technology is really in their favor, right? And I think it's a process, like it's a process iteration until people find the right use cases.

So I guess that's a good transition to the future of the technology, future of the product. If I recall correctly from the No Priors episode, I believe you said 40 employees at the company today?

Yeah, we have about 40 and 50, somewhere around that.

Yeah.

So I'm just doing a little back of the envelope math. If you're doing $35 million in annual revenue, that would seem to be plenty to cover the employee base. And so one would then assume that a fresh slug of capital would be going into a lot of compute, maybe like licensing of underlying content as well. Like, what's the kind of, you know, use of proceeds here? And what is sort of the next big act for HeyGen?

So when we initially founded the company, I think you touched a little bit about when you look at HeyGen as a customer, as a user today, you saw it mainly as featuring the avatar videos. That is the main power of the HeyGen platform today. But we actually never see us as an avatar company. What we really, really want is solving the problem of generating a video for businesses. And if you look at really the way how we solve that is sequencing in the way that we want to solve the A-roll problem first. What I meant by A-roll is mainly the human spokesperson, the avatar, the actor piece. I think there's a big piece is still not solved in the industry is actually the B-roll, right? All this background music, transition animation, stuff like that. You know, I think the next act for HeyGens, 100%, we want to continually improve the quality, improve the engagement on the A-roll piece. But also, I think putting a lot of investment on B-roll generation is another thing that will be critical for us to achieve our mission to generate the entire video end-to-end. I think that involves mostly probably model training, investing in the talent, product development, stuff like that.

So maybe taking those 2 things in turn, in terms of improving the quality, is that something that you have to scale up a base model for? Is there a scaling law for avatar video generation, or is it more about a reinforcement learning approach where it's not so compute intensive, but more about actual user feedback? What is the kind of key input to improving quality on the current margin?

I would say it's a combination of both. It really depends on what other problem we are trying to solve. For some of the quality problems, the more data you are able to collect, the more edge cases you are able to cover, and your model naturally gets better performance. But I think I want to bring up another point, is that solving a video generation problem is not only about solving a mathematical problem. It's very different. And how would you modularize the conversation flow problem? In the actual AI training. That is unsolved yet. Being able to identify those areas and engineering in a way that having AI model can actually help to solve that and figure out a mathematical way and also apply that into the product, I think it's another big topic about the future development as well. Generally, I would say That's a combination of more data, more compute, more model breakthrough, and also more product innovation there.

On the B-roll side, right now you have kind of the studio-type experience where you can go in and edit stuff and kind of move it around a timeline a little bit. Is the sort of more advanced version of that starts to kind of encroach on Adobe territory, the more large-scale foundation model maybe starts to go more in a Sora type of direction, maybe it's both there too, but I'm kind of wondering like, how do you plan to tackle the sort of everything else after the avatars?

So I think you mentioned 2 things about one is the traditional timeline editor. And the other one is more like end-to-end text-to-video generation. I think the approach we are believing is different from these 2. First of all, I have a very aggressive view and a strong point of view that timeline editor will be gone. in 5 years. But today, the entire video world was defined by a timeline editor.

Sounds like—

Probably 20 or 30 years ago, right? But the whole point of timeline editor was because camera is expensive. And because camera is expensive, you need to film it 20 times or 30 times, and then you put it on the timeline, say that's the best one. But when the foundation or the starting point is changed, where you can generate the footage on the fly, The editor experience will be massively different. Do we need a timeline anymore? I don't think so. And I think in order to enable the, like I said, the rest of the 99% people be able to create content, timeline editor is actually a very big learning curve for everybody to learn about. And there must be a new paradigm in terms of editor exist that can help to embrace this new technology generation paradigm. That's one. And the other thing is that HeyGen focuses a lot on the business videos. And when we look at business videos, what do they need? They need quality control and consistency. We found that purely you can either solve it with the end-to-end text-to-video model way, or I think what the approach we believe in is that we build a precision engine. To really understand what the brand needs. And then in this case, you see the avatar generation, A-roll generation is one component behind it. And then voice, music, voiceover, B-roll, and stuff like that. And the approach we really believe in is that building this optimization engine and combined with the editor to enable the experience down the line.

So do you expect somebody like me to sit in front of a screen and say, that's good, but replace that background with a green park background instead of the fall colors background and just wait a few seconds and then watch a new version and just kind of talk to the computer to iterate? Or like, what does that actually look like from a creator experience standpoint?

Yeah, I mean, like forward looking a little bit, let's say Nathan, the AI got to know some of you, your past videos that you have, it can naturally just learn about who you are and then how to regenerate your talking videos like that. I don't think it has to be that you talk to the computer to switch. I mean, it could be anything. It could be a UI toggle or any stuff like that. But being able to put a person in a different scene, it is very possible today. Just the fact that does it actually match the quality that we want in a professional setting, professional quality? And the problem there is actually the lighting, right? So not about the background removal and background replacement. And how do you— let's say I put my view right now and put it into a beach, you know, and the lighting actually feels weird in that way. But any approach you can solve in order to make that happen. And then next question would be, you solve the static version of the lighting problem, then how about the dynamic lighting? You know, You're moving your gesture. How do you make sure the whole thing is like, feels natural and realistic and engaging? I don't think we are very far from being able to generate A-roll footage in any of the settings that you mentioned. I think that's pretty possible.

So if I am understanding correctly, it sounds like the vision that you have is for, you know, leaving flexibility at the level of the user experience, a kind of Swiss Army knife of a lot of different AI tools under the hood that the user maybe doesn't even know about in detail in terms of what model is being used or whatever. But it sounds like maybe a couple of core generate the footage type models, and then a lot of auxiliary models that do specific things like change the lighting or do this, do that. And those get orchestrated through some user experience that presumably is high-level qualitative. You get to say what you want. Maybe even it learns from your past stuff, but under the hood, it's doing an intricate manipulation with a lot of different tools. And that would be, as you said, that's in contrast to something like Asura. You're not trying to create a general world modeller or an intuitive physics engine, but instead just all these kind of things that do their job really well. And your kind of macro job is to make them work together.

Yeah, I want to add one thing to that though. When we talk about the different pieces of the system, right? The orchestration engine has a different model behind it to empower each of the features or functions as a tool, let's say. I do believe there is a role where a single model can solve a lot of these different tooling problems. I'm not entirely sure what would that look like. But with the technology evolving this fast, I think we have a way to really nail down to like a couple of major models behind the scenes to enable that. And then another angle, I would say, we talked about lightning just now, but when we really look at it, the lightning problem was defined in the camera world, right? Because we use camera. Okay. We need to pay attention to lightning. That could be a problem. And then if there's a problem, we could actually fix that in another downstream. Video editor. But when we really look at the new paradigm, lighting may not be a major dimension we need to talk about. It could be the dimension that, hey, make it more engaging. It could be another dimension about make it more dynamic and make the conversation flow more fluently to express the excitement about the speaker. I think those dimensions will be new and could be possible with a new paradigm. of this no-camera world where you can actually generate the entire thing.

Yeah.

Yeah, cool. That's interesting. I think this, the dramatic simplification of the underlying architecture is something I've personally experienced with Waymark. I don't know if you've seen what we do, but it's another kind of text-to-video generation experience, except we are basically making TV commercials most of the time. We partner with a lot of cable TV companies, broadcast TV companies. And so the constraints are super specific. It has to be exactly 30 seconds, yada, yada, yada. And I started the company actually, but now I'm the AI R&D person. And the simplification that we've had over the last year from an earlier form where it was like, even just to figure out what images to choose out of a user's image library used to be an incredibly gnarly thing where we'd be captioning, but the captions were super generic and we'd use an aesthetic evaluation model to try to figure out which ones look good. And now that's largely just been reduced to, ask one of the latest large multimodal models which images are good to use, and we get better results actually from that. So it's a remarkable trend that these Swiss Army knives are seemingly evolving toward fewer features, but they still work a lot better. I wanted to move on in the last few minutes. I know we're going to be out of time before too long. Certainly anything else you want to talk about in terms of technology future direction is welcome. But I also wanted to get into the trust and safety stack because that was actually the thing that most impressed me about HeyGen, and it really stands out in the broader market. So I'll just tell you what my experience has been. So I was a GPT-4 red teamer. That was kind of how I initially got into the headspace of testing AI products for risk of misuse and abuse. And more recently, I've been noticing that a lot of products are being stood up, especially like agent products, but also these sort of video generation and voice generation, especially with cloning type products. A lot of them are being set up with no guardrails at all. You can go into a lot of these products. I've cloned Donald Trump and Joe Biden and Taylor Swift's voice on many different platforms at this point, including some that allow you to make a call to an arbitrary number and just say, like, solicit a donation from this person or whatever, right? You get literally no constraints at all. It's kind of mind-blowing that people would set something up like this and put it out there and have no guardrails. In contrast to that, HeyGen has, I would say, honestly, the best integrated security measures that I've seen. And I would love to just get your kind of take on how you guys have thought about that, how you've prioritized it. I think a lot of businesses feel like that'll slow us down or it will annoy users or, you know, for whatever reason they're not doing it. But you guys have demonstrated that you can both build a business pretty quickly and to a significant scale while prioritizing that in a meaningful way. So I appreciate that and I want to hear more about it.

Yeah. And maybe before Josh comes in, Just to reinforce that, I was also very interested in this video creation space with avatars and there's a few teams going at it, right? And in my head it's like, hey, this technology for sure will exist and customers want it and customers will use it. And okay, the technology is so powerful, there's a huge risk of not using it well. And the thing that stood out for me for HeyGen is this was central to their strategy. It was like, hey, we're going to be the most trusted brand. So Like, Joshua is going to come in and talk about the details, but one of the things that excited me most about the company is understanding how important trust and safety is.

So I would start with like this. We never see trust and safety as a factor to slow us down. And I always see trust and safety as a critical piece of the business. It's actually part of the product. Every time we roll out a new feature, We have a new experience we want to share with the customers. There's always a session of trust and safety. How do we actually safely roll out to our customers? And I would say trust and safety is critical to our business because we are serving some of the largest companies in the world. And I can share a little bit how it is being done today and how we really think about it. So first of all, about creation, we don't allow any political, or celebrity figure to be created on HeyGen. And every single avatar or digital twin created on HeyGen has to obtain first-party consent in a video format. We also do a random-generated dynamic code verification as well as human review operation in the back to make sure the system is not being misused. Another thing we do is that For every single video you are creating or localizing on HeyGen today, we have a combination of our AI model and human reviewer together to really make sure the content you're putting out there complies with our policy in terms of the product. Like, for example, no hate speech, no harassment, no fraudulent, no misinformation, and stuff like that. And those are some of the guardrails and product design decisions we put together I hate ChatGPT.

It is really well done. I went through it and with the goal of actually trying to break it and never did end up breaking it in the most flagrant ways anyway. I was able to get a little bit of like offensive speech out of it here and there. But in terms of like the big things where I upload the Trump video or the Biden video and try to get the avatar from a public figure, I was not able to do that. And I did kind of encounter several different, I would say, I don't know if you would agree with this characterization, but It felt to me like a defense in depth strategy where there were multiple different checks happening. And I think I systematically found quite a few of them, ultimately landing on the final one being a human review that's just like, yeah, sorry, we're not going to allow this. When you talk about celebrity faces, this is one interesting question I had, and I wonder if there's something here that others could learn from. Is there any special tech? Did you have to invent the technology? How are you doing that? Who's a celebrity? How How many celebrities are there in the world? Like how many celebrities are not allowed on the platform? And is there anything that you're tapping into that others should be aware of that they might also be able to tap into?

Yeah, so I'll put it this way. So first of all, particularly for, you know, celebrity, I think there were legit use cases for that for sure. And I guess we just don't want to expose the free-form creation process in that case. And they do have to contact us and then we can involve, you know, with our operation team to help them to create the avatar or digital twin that way. So this is 2 different approach, you know, but being a little bit more heavier in terms of process that way. The way how we do it is that obviously we work with some of the vendors who had a, you know, a celebrity or just like generally the API that provide us a database. We also constantly having our own team. To update database to ensure that we're having well coverage.

Interesting.

Okay.

So there is a company, you're paying another company to provide that service?

Yeah. Yeah. Okay.

That's interesting. I've been looking for one and I have not really found one. So I'm going to have to go do a little more digging on this. I mean, the reason I want to promote this is that I see so much product surface area out there, not HeyGen, but most of the rest of the space, honestly. is so wide open to abuse that I've been reluctant to even disclose it at times because it feels like just irresponsible to put more attention on it before it's solved. But then when I talk to the developers and say, like, hey, here's what I did on your product, just FYI, this looks like a problem. A lot of times they come back to me and they're like, well, I don't really know what you want me to do. How am I supposed to detect every celebrity voice that somebody might come through with? So, yeah, I don't know, maybe I'll have to see what I can find out about that, or I don't know if you can leave me any hints somewhere, but I definitely want to popularize this. If there's a good solution for people to adopt and all they have to do is pay for it and use it, then I'm definitely keen to figure out what that is and try to promote it. Cool. Well, this is fascinating. I think the big takeaways I have from this are, again, it's a common theme. This is also early. Don't look at just like an avatar today and think this is the end of the line. Not only are they going to get better, but also they're going to get faster, they're going to get real time, they're going to get more responsive, they're going to get more contextual. They're going to create a lot of different kinds of experiences where the paradigm of what is a video that we have kind of intuitively developed ultimately maybe gets blown wide open. And it's like, as always, the future is like a little weirder and maybe cooler than we initially dare to guess. So I think it's another great reminder of that. And I do appreciate the care with which you guys have layered in all of the different safety mechanisms. I think that is something that truly people should come like study your product, look at how it's done, and take inspiration from that back to their own products. And I Don't say that lately. I've tested a lot of things. So to come out on the top of my personal power rankings there, I would say, is actually a legitimately real accomplishment. Anything else you guys want to cover before we break for today?

I think we covered the most important things.

Okay, cool. Well, I really appreciate this, guys. Joshua Xu, founder and CEO of HeyGen. Victor Lazarte, general partner at Benchmark. Thank you both for being part of the Cognitive Revolution.

Thank you for having us.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.