Next-Gen Technology and Authentic Human Avatars | AvatarOS | PI Apparel

So my name is Isaac Bratzel. I'm the founder and CEO of AvatarOS. We're a tech startup backed by A16Z and NVIDIA. We're focused on empowering extraordinary avatars that can scale like software. So my field of expertise is digital human avatars, and that's what I'm going to talk about today. So if you're wondering why I'm here talking about avatars at an apparel conference, hopefully you can guess. But my premise is basically that these technologies are very connected and will be more so in the future. And I believe digital humans will be crucial for the fashion and retail industry in the near future. Fashion and retail are interesting case studies for digital humans. And the case study— and the specific needs of these industries, I believe, have been vastly underserved to date. The good news is that the technology exists today to create tremendous breakthroughs that will generate enormous value in the years to come. So I'll mention the key technologies and how this really is going to apply to digital fashion specifically. And then as I go through the talk, I'm going to focus more on the technology and what's going to be possible in the short future. But you— everyone here knows more about fashion than I do. So I'm going to focus on what I know about and why I think it's relevant. And you guys can connect a little bit more of the dots. And then I'll leave a little time at the end for questions if there's anything you want to ask. So the key innovations include dynamic targeted advertising, personalized models and garments, on-demand production, made-to-order items based on customer interest, eliminating the need for physical stock until purchase. These are things you are all very well familiar with. Interoperable digital assets is a big one with seamless integration across video games, AR, and other platforms. And interoperability here really refers to the idea that it's not feasible to have multiple production pipelines for digital assets. For every different use case, for WebGL, for a mobile game, for a video game versus video production, high-end production, and then assets that actually are digital twins and meant to go into print. These are all very different pipelines for digital assets. And expecting customers to maintain those or create different pipelines for each one is just not a practical thing in the future. So the benefits of all this is enhanced user experience, interactive engaging shopping and gaming experiences, cost efficiency, reduced production and inventory costs, and innovative advertising seamlessly integrated ad experiences within digital environments. So to get us started, why aren't digital humans already making more of an impact in the industry, and what can change and will change in the near future? I put together 5 premises that I really believe that are the reasons why this isn't the case yet and what will change soon that will make it possible. The first premise is that current avatar systems don't nearly scratch the itch in industries that care so much about refinement, authenticity, and quality. The second premise is that image generators have issues with fidelity, consistency, quality, and control. And in the end, users are stuck in two dimensions. Premise 3 is that broad adoption of tools such as Clo3D and Marvelous Designer mean 2D patterns are able to be built out digitally in 3D with a high level of sophistication. Sophistication and accuracy. Premise 4 is that access to quality avatars and high-quality rendering is a huge blocker to broad adoption of digital fashion, especially for advertising, but across the industry more broadly. And premise 5 is that interoperability creates a huge hurdle and that interactive experiences and games require a second pipeline for wardrobe assets to create optimized assets. OK, so my conclusion is that apparel is for humans. It's very specific to us individually. And digital apparel needs humans in order to ground the users in the experience. Limitations to stylized, simplified, and unconvincing digital representations of humans in apparel leave users unconvinced, unimpressed, and unsatisfied. So to date, I think digital fashion has had a mixed response. And I think there's obvious reasons for that. But this often happens with technology where the first iterations of it create some view of the future, but also a lot of negative feedback because they're not quite where they need to be. So today, I'm going to focus on where we're going that will alleviate a lot of these issues. OK, so the reason digital humans are so hard— let's get into this a little bit. When we're talking about really high-end digital humans that are actually trying to recreate the fidelity of a real-life human in a digital space, there are two really key problems that come into play. The first is the uncanny valley, which I'm sure you're all familiar with. And the second is exponential complexity. The uncanny valley is a well-documented term for the negative response humans have to representations of humans as they get closer to real life. In other words, when it's almost real but not quite, it feels creepy, right? It's better off being stylized, or it better get all the way to feeling really authentic. Otherwise, my human response is going to be very icky. Exponential complexity describes the compounding effect of high-quality avatars on every aspect of production. In other words, the quality of animation has to match the quality of the rendering and has to match the quality of the simulation and every aspect of it so that when you're just improving the quality of the avatar, you're not just improving— like, you're not just increasing the cost of that one area. You're actually increasing the cost of animation and rigging and simulation and rendering and cloud provider costs and every step along the way. And this is why costs tend to explode to the point that most people don't really realize. One note I have here is that She-Hulk, which is a TV show if you haven't seen it, spent $75 million for 3 episodes. And that is almost entirely because it has a digital character as the star instead of just a regular human. All right, so at this point, I'll give you a little background on me so we kind of know where I'm coming from. If we go back about a decade, I was the head of 3D at a company called Ubisoft where we created Amelia.AI. Yeah, almost a decade ago, we were working on interactive AI avatars. Go figure. Amelia was a cognitive agent who communicated and problem solved in real time. She was backed by proprietary natural language-based cognitive AI, and she had over 9 million active users. She spoke 108 languages and had thousands of integrations. She boasted a great success rate, resolutions of 93%, and 91% satisfaction for sales, and had over 4 million conversations. And she's still going strong today. In fact, they rebranded to Amelia.ai. If we fast forward just a few years, I left Ubisoft and joined a startup in Los Angeles called Brud as their chief design and innovation officer. BUD was a stealth startup backed by Sequoia and had the radical idea to kind of hack celebrity and hack what a human is and have a digital character that invaded typically spaces typically reserved for real humans. We created Miquela, who you may have seen or may not. She's a digital character who had an online presence and a narrative that fans engaged with at a remarkable rate. Miquela grew her online following to over 3 million Instagram followers, 3 million TikTok followers, and 250,000 YouTube subscribers and over 50 million streams on Spotify. So Miquela was truly a global phenomenon and a virtual star. She partnered with the likes of Calvin Klein, Prada, Samsung, and many more and was even named one of Time Magazine's 25 Most Influential People on the Internet in 2018, which is not bad considering she's a digital character who didn't even exist a few years before. I'm going to play a quick video so those of you who are not familiar can get a sense of what Makayla was about. And it's better with audio, but you get the idea. And since there's no audio, I'll just talk through it. It might be a little more engaging. So we did all kinds of things from brand partnerships, advertising, music videos, in-game. We did a full 5-minute music video during the pandemic in Unreal Engine that was all interactive and had all digital environments, et cetera. A lot of the things you'll see would be computer graphics, which is generating traditional VFX, generating computer graphics that blended VFX of real-life footage with CG assets. There's really no limit to what you can do. The only problem is the cost, right? The cost of doing this is not very practical. So we learned so much in the process of creating Mikayla. Oh. We were often surprised by fan reactions. When it came to upgrading Mikayla's appearance, we worried about fan response and thought about how we could weave in changes to her visual appearance to her narrative. We were really concerned that these fans that cared so much about Mikaela and the story would kind of reject this idea of her upgrading over time, right? And that was time wasted, it turned out. It turned out that as we did that, fans were totally up for it. They didn't give it a second thought. They didn't miss a beat. But at other times, fans would rant and rave because they were sure a shadow looked like fake. It wasn't quite right. And I swear to you, at least half the time, that shadow was like a photographic element. It wasn't even computer generated. So the learning was really that it doesn't really matter so much if something's real or fake. It matters whether it feels authentic. It matters whether you put the user in a place where they're experiencing this and they're along for the ride. If you think about a sci-fi movie or something, like, once you're in that movie, you accept the reality of the story it's telling you. And when something feels fake is when it takes you out of that reality. Not whether it's real, true to our existence in the regular world. It's whether it's true to that existence that you've created. In this online digital world. So that was one of the key learnings we took from Miquela. And that learning is all about authenticity, by the way, which I'll talk about a lot more. So the biggest problems, as I've alluded to so far, with both Miquela and Amelia were the immense cost of production and the constant uphill battle of animated content that could hit a quality bar sufficient to create authentic connection with fans. In Amelia's case, it was incredibly difficult to bring to life a character with only text input, especially back in 2016. But the thing that always stuck with me is this idea of scalability and deployability. Amelia, while she was limited, was software. We could deploy her as an agent, as a cognitive agent in multiple roles. And there was no more cost per second of animation. There was no significant cost of replication. She was actually able to scale. And that scalability was something that I've had in my brain ever since. As we were building Miquela and now with our company AvatarOS. OK, so if you take Miquela as a case study, just so we all have a sense of how much some of this stuff really costs in the real world, we see that the cost of a high-end avatar just alone ranges in the six figures. And it can be a pretty broad range, but it really just depends. And depending on your production pipeline, even if you're just talking about stills, but especially if you're talking about animation, you're looking anywhere from hundreds of thousands of dollars to millions of dollars a month to consistently produce high-quality animation. Now, if you go to something like Framestore, Digital Domain, MPC, any of these fantastic companies, those numbers are going to skyrocket even more because their budgets and their pipelines are built for feature film. And they're really not even close to feasible. They weren't even feasible for us when we were doing this. We had to build our own team and create these pipelines from scratch. Digital humans are the most complex and expensive aspect of development in an industry where costs are exploding. It can cost hundreds of thousands of dollars for AAA blockbuster characters. Especially hero characters, and even more to implement them and maintain them. The other problem is that these avatars, once built, are typically built for one product— one project, sorry. And after a few years, they're obsolete. You rebuild them from scratch and start again. In every VFX production house and game studio, there is entire art teams— concept, modeling, texturing, grooming, cloth surfacing, lighting, look dev, rigging, animation, simulation, rendering, compositing. Those are all different artists in different roles that touch the digital human throughout the production. And there are 4 loosely defined examples in the digital human marketplace that strive to solve this problem in different ways. And none of them so far have really created a complete solution. So we have studios, like I mentioned, that solve for quality through brute force. They solve the uncanny valley by embracing exponential complexity. They throw artists at the problem. They build bigger teams. And they have million-dollar budgets. Then there's stylized avatars who avoid the uncanny valley entirely. They say, let's not try to deal with that problem. Let's build it cheap. Let's make it scalable by stylizing and simplifying the avatar. We have character creators, which are great tools and have built— been very successful in terms of democratizing technology. So things like MetaHuman Creator, Character Creator, Daz 3D, other ones you're probably familiar with. The problem is that by their very nature, they are generic. That's how this software works. And that never really gets us to the point where we can create individualized authentic avatars that are really high quality. And finally, we have AI-powered avatars. They demonstrate great potential, but to date come with extreme limitations, like Amelia did back in the day and like we see today with a lot of the shoulders-up, head-facing 2D avatars that are AI-generated. OK, so just a quick touch on the digital human avatar market. It's projected to reach $527 billion by the end of this decade. So this isn't a small niche industry looking forward. It's one that's projecting tremendous growth and growth that requires much more scalable solutions that exist today. Make sure I didn't miss anything. OK. OK, so just to recap, the problem is that complex and— the problem is complex and unscalable processes, exploding costs, long development cycles for lifelike digital humans. With no ability to lower production quality due to the uncanny valley. So with that, I give you our solution, which is to blend human authenticity with technological efficiency. The solution we have is AI-powered avatars at the quality of real life. We can solve the recipe for real digital humans with data and code by training on real humans in motion and creating individual character identities. This results in AI-powered avatars that can scale like software, cross the uncanny valley, and have no significant cost of replication. Obviously, that's digital Isaac, and he's saying things that you can't hear right now. And here's a look at our patented avatar training process. So what we do is we use volumetric 4D motion capture. And this is a lot different than the motion capture you may have seen on video games and movies where there's a person wearing a suit and a bunch of dots. What that does is it creates a triangulized estimation of joints. So if I move my arm like this, it's going to say, based on these dots, we're going to solve for where's a point for your shoulder, a point for my elbow, a point for my wrist. And that's the motion this avatar is creating. And what we've seen is that is completely insufficient for really lifelike deformation. What we do is we have a bunch of artists go and now use that motion as a reference and create more complex animation. What we're doing is using volumetric motion capture, which actually solves for the entire volume and shape of my body as I'm doing these performances. Then we can go in and create ranges of motion that create all the deformations my body can create. And we solve for that in the unified mesh. We solve for joint placement. And then we solve for deformation that can line up against the standardized character rig. What that means is we can create deltas. And the delta is a difference between what the character rig expects to see based on linear skinning and what the actual ground truth product is. And those differences are baked into a machine learning deformer that in real time can correct the deformation of this avatar to look like the real-life version. In other words, we can have an interactive avatar that matches the ground truth performance that was captured in 4D. And this is really exciting because in the past, 4D was used as reference. 4D was used for a specific canned performance. It looks amazing, very difficult to use, and completely static. It's basically like a 3D video. That's all it is. And now we're turning that into an actually interactive and dynamic avatar that can be scalable and cross the uncanny valley. OK, so simulation and training. Lifelike avatars allow for lifelike simulation. Once you have this avatar that can perform and act in a fully lifelike way, anatomically correct, et cetera, you can now simulate accurate garments on top of that that will look lifelike and believable as well. And you can use the same process that I just showed you to train machine learning solvers for this wardrobe. So if you look at the bottom left here, what you're seeing is a very simple example and test for a machine learning solver. On the left, you see linear skinning. What that means is there are 3 joints there, and there's a mesh. And you basically do what's called skinning or rigging those 2 things together. So as you rotate those joints, the mesh follows along in a mathematical simple algorithm. Now, if you look at the far right, what you're seeing is a simulation. You're taking a simple object, and then you're turning this mesh into cloth and doing a very crappy simulation. Again, this is just a test. You can imagine it's an arm, but it's just like— this is just a very simple test. So you're seeing, OK, this is what the simulation should look like. So imagine this is something you spent hours on that you've approved. You've gone through revisions, all these things. And that's what you want that cloth to look like. Now we can see the traditional model of machine learning solver in orange that is trying to deal with these complex rotations, et cetera, and failing miserably. And now you can see ours in blue. And what you can see is that in real time, based on any motion that we create, we can solve for deformation without simulation on the blue mesh that matches the ground truth of the green simulation. And that's what basically the magic of machine learning solvers can do. So these things would match up entirely with the character that you've trained on and create really exciting results. OK, so back to AI avatars for a second. Deploying AI-powered interactive avatars can be flexible and scalable as well. Here we demonstrate a modular and customizable tech stack where ChatGPT can analyze audio, video, and text input and generate a response. ElevenLabs can generate character-specific voice. And NVIDIA Audio2Face can convert audio to facial animation curves. And this tech stack is modular intentionally so that we can connect any LLM and any animation system to drive these characters and build them into custom implementations. And if I could play the videos, basically what you would see is on the left, we're going to talk to this avatar and type it in. And we're going to get vanilla responses out. It looks like a puppet mouth. There's no expression. It's a little bit dead. This is what you're going to see from most chatbots over the next little bit. And on the middle, we're able to take those inputs that are very simplified and based on the character training identities that are specific to this character, generate a lifelike and believable performance, emotional and even the way I speak and the way I move that matches the training that was input from me with less than 3 minutes of training data. And on the right, you can see how we're creating— generating a smart idle system and doing simulation, etc. And here you can see the result. It's the quality of real life and enable practical utility at scale. OK, so while it's far from perfect and we're still in the early innings, but with less than 3 minutes of training data, being able to create highly accurate character-specific interactive avatars is incredibly exciting. And we're really, really excited about our progress and the implications that it will have. As this technology improves, it will truly become unlimited as the cost for generating authentic performances functionally goes to zero. OK, and this technology is not limited to digital replicas of us. So far, what I've been showing you is how I can take a specific person bring in a kind of expensive training process, capture them, and turn them into a digital double of themselves. But the learning for this is not strictly limited to that individual person. We are able to use our learning from human motion to generate novel data sets for unique characters. Here you can see the process by which we generate an identity for a fictional character and art direct the new range of motion leveraging supervised learning. We can artistically intervene to ensure accuracy or insert our own desired outcome to give the avatar unique characteristics and control how they specifically physically move, speak, and act. Then we retrain the results for further fine-tuning our model. And here's a couple of examples of that. I don't know if we'll see the videos play or not, but— Interactive and dynamic content powered by AI will be the next wave of new media. We support cross-platform integration of AI within leading game engines, enabling truly interactive avatars that can produce unlimited unique content. OK, so again, these were not real humans that were captured or trained on. These are fictional characters and the learning models applied to them and artfully deciding how this character moves and speaks and how it should interact. And by using physically accurate materials and procedural processes, we can introduce a level of simplicity to digital fashion and apparel that till now has been lacking. In other words, creating really high-end and photorealistic quality to date has been a really complicated artistic technical process. But that doesn't need to be the case anymore. There are physical attributes that each material has. And once you solve for avatars and lifelike renderability and deformation, rendering a digital avatar in a digital studio could be as easy as selecting the pattern, selecting the texture, and selecting a pose, and that everything else can be procedurally done. So this is an example that we created with our digital model Mei, where we— do just what I just said. You select the avatar, and then you select the pose and apply the pattern as such. So what we're building is not a character creator, which is, as I mentioned before, is by its very nature generic. And we're not really an image generator, which is problematic in its own ways. What we're creating is an operating system that allows scalable processes for lifelike digital humans and by extension, digital apparel. Each character we create is unique and artfully crafted. We know firsthand that quality and authenticity matter. And the possibilities for these avatars across games, film, social, retail, fashion, advertising, education, and many, many more are truly unlimited. And just like each avatar is unique, every avatar use case is unique. Our proprietary AvatarOS software is modular and node-based. It's built specifically for flexibility to create custom avatar systems and solutions. In other words, we want to allow you to create the system that your product needs for your customers, not an end-all be-all platform that just works magically for everybody because every single use case is so different. I'm going to try to wrap this up quick so that we have at least have a minute for questions. But here's one more example of a custom avatar system powered by AvatarOS with a cloud-based render engine and an intuitive text interface. So you can see the ability to select different avatars, select a wardrobe that is interchangeable and interoperable, render via NVIDIA Cloud GPUs, and even prompt based on stored animations. So if this is something that you could have a bunch of animations that your client has or you have or could be supplied by AvatarOS, et cetera, and you could use very intuitive methods of implementing AI to access these animations and poses. You probably can't read that. It's basically prompting me to add an HDRI background. OK, so just to wrap up on us, we're backed by A16Z. We're very young, but we've been accepted to the Speedrun Accelerator, and we're part of the NVIDIA Inception program as well. We have patented proprietary software that is unrivaled in quality and innovation, and we're projecting $1 million ARR by the end of this year. We've already done exciting projects, as you can see here, with Burberry, Balmain, and Alexander McQueen, NFL, and Major League Baseball, and a few other specific strategic partners. And moving forward, we plan to have access to 10,000 licensed unique ranges of motion captured in full 4D volumetric motion. That means this is going to be able to scale and improve in quality dramatically over what you see today. So take 3 minutes of training data and multiply that by a whole lot, and you can see not just the amount of data, but the quality of data will really improve. We're building with a long-term scalable approach, and we're not limiting quality to the constraints of today. In other words, we're not building something that's built to crunch down into WebGL or some application that's going to make the quality look bad. Like, everything looks great, but if the end result on the user is not great, it doesn't matter. What we're betting on is that the build-out of cloud GPUs by NVIDIA and others is going to allow us to stream pixels to final destinations. So that you don't need to modify your build to be WebGL compliant or anything else. You're actually going to be able to stream from Unreal Engine or NVIDIA Omniverse or another platform straight to your destination and have that be free of lag and actually optimized enough to be lifelike and believable in quality. OK, so just to wrap things up really quick, some final thoughts. There's nothing in our universe that we respond to the way we respond to another human. The growing impact of the digital world and the acceleration of AI are outpacing our ability to generate that authentic digital human representation in the digital world. But the recipe for digital humans can be solved with data and code. And it can reach a scale that has no significant cost to replicate. Interactive digital humans that are perfectly lifelike, that can scale like software, that can integrate everywhere and anywhere, individually embedded with artificial intelligence and natural language, this will change the very nature of our digital world. The question is not if, but when this will be possible and who will build it. All right, that's it. So I can take some questions. This is me, by the way. You can scan the QR code if you want to check out my LinkedIn and reach out or grab me after the talk. But I think we have almost zero time. So if there is any question, I can take it really quick. Otherwise, I can let you guys get on with it. But thank you very much. It was a pleasure.