Practical, positive uses for deep fakes

I think this is really an exciting time where creativity is basically unleashed both in synthetic media, human faces, person that does not exist and also prompt generated images and styles and all of these kind of things are coming together and people are getting used to basically consuming stuff that is automatically generated and creativity is basically changing. People might be frightened and I think that we will use it for good and for misuses and we'll adjust as going forward. But technology is moving at a fast pace so of course there is. It's a whole spectrum and we're exploring this spectrum and I think it's very interesting place to be.

Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly IO we deploy our app servers close to our users and you can too learn more at Fly IO Foreign.

Welcome everyone to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing Chris?

I am doing okay. Although I have to confess this. A few days ago I took a fall while rollerblading and so I'm on hydrocodone for the pain. On that I could say absolutely anything.

I'm really interested in this conversation even more now.

One thing I'll finish by saying is I discovered that when it comes to rollerblading, I am a deep fake. I am not a talented rollerblader. I'll leave it at that.

Yeah, well, speaking of those deep fakes, we're going to dive into much deeper conversation around this topic that we've mentioned. So Chris, we've talked about deep fakes a few different times. I think it's come up. I think the wider, even non AI people are aware of this technology and some of the things that have been created. But today we're really privileged to have with us Lior Hakim with us from our one who is the CTO at hour one and he's going to be talking to us all about this technology and what they're doing with it. Welcome Lior.

Thank you. Thank you for having me. It's a pleasure. And we can dive right in and speak about AI and deepfake at large and what we are doing.

Yeah, for sure. Maybe Just starting with that. When someone asks you what is a deep fake, you mentioned in a conversation, what is your explanation of what a deep fake is?

Yeah. So basically what we're trying to do is virtual human and to humanize the connection and how we communicate with machines, basically because today we are very used to text pages and then frozen video. And that is changing and interactions with machines is actually in the future will be different than what we experience now. And defects or other technologies, synthetic media or other names that are named like this is basically some bridge point of how we interact or ingest information and communicate with machines. So I think that's exciting.

Yeah. And I guess some people are probably aware of things maybe they've seen on the Internet where, you know, I think there have been ones with like Elon Musk or other people where people have created a video of Elon Musk saying something. But it's, it's a, it's a synthesized video. And I know that that's maybe something that comes to my mind when I think of deepfakes. Like I was saying, there's many people even outside of the AI community that understand that AI is able to create these very powerful and compelling videos, whether they're for misinformation or for good, good purposes. AI is sort of starting to impact people's like content that they're viewing on social media or wherever. How have you seen that develop and from your perspective, how do you see the trend and how AI is influencing like the creative side of what people are doing with video and other things?

I think this is really an exciting time where creativity is basically unleashed both in synthetic media, human faces, person that does not exist and also prompt generated images and styles and all of these kind of things are coming together and people are getting used to basically consuming stuff that is automatically generated and creativity is basically changing. I think that there are of course, with every technology that comes about and you people might be frightened or not used to yet to these types of things, and I think that we will use it for good and for misuses and we'll adjust as going forward. But technology is moving at a fast pace and later on we will talk about of course what we are doing. But I think that a general thing to consider is maybe Personas in the context of deep fake. So we have those Personas. I don't fully subscribe to the notion of a fake and real because I think me in real life and me in social media and other stuff like that might be different. I might present different things. So I Think it's a whole spectrum of uses and misuses. And definitely some things can be unusual. Sometimes maybe I would like someone else to be able to use my Persona for some cases, like, I don't know, people on commercials or landing my image for a character in a movie that I'm playing, having the writer's text speak it and have the director direct me. And then in other cases, I might be more frightened if something that I didn't meant to happen, I see myself speaking something that I don't know. So of course it's a whole spectrum and we're exploring this spectrum and I think it's very interesting place to be.

You know, the way you started that explanation was fascinating to me because I think so many people are introduced to the topic of deep fakes by some of the nefarious things that get popularized in media reports and such news reports. But you talked about it in terms of the way we're interacting with computers, sort of that user experience to some degree maybe. And I found that really interesting because though I'm in the defense industry now, I spent over a decade in the creative digital marketing space and we were all about Personas. And I don't think I had really adjusted my way of thinking about deepfakes to think about really focusing on the interactions versus some of these other ways that we've seen, which tend to be more on the negative side. So could you talk a little bit about that for a moment? Because you've kind of reset my perspective. How can deepfakes take us forward in the years to come in terms of how those interactions with automated systems play out? What's different and what should we expect as the normal. As we're looking at some of the possibilities over the next few years?

So I think your likeness, the way you look, the way you sound, the tone of your voice, the style, your gestures, everything is kind of like your set of skills that you're regularly put for work for other people. You go to work, you give your content, and then those types of traits, you will be able basically to digitize them and then put them to use in the context that you feel is suitable for what you want. For example, you will be able to capture yourself, digitize yourself like an avatar. You can think about, I don't know, Bitmoji. Okay, you directly design the avatar like yourself, and then you put it to use not as yourself. And the same thing can happen with real life capture video or with versions of yourself, filters that make you more attractive or more happy. Or if you don't want to put makeup every day. You can just hop into a meeting or sales presentation looking as your best self from your home, with all the work from home and all, all those kind of things. I'm just giving a few examples not to. To be too abstract.

Well, thank you. That whole thing, face for radio here. So, yeah, I mean, that's. That sounds fantastic.

Like we're sitting now. You know, we're casual, and then people can sit, see us talking in a nice studio, not just listening to audio and enjoy everything without the necessary needs for us to cut our head and shave and do whatever we do to present ourselves. So our likeness, our tone of voice, and how we present ourselves, our Persona, how we perceive ourselves basically can be digitized and then put to use. And to. If we can control, we can of course control it by ourselves and have maximum control over the content that we deliver through our digitized character. And also we can lend those character to other people to put content through us if we think we can affiliate ourselves with that content, if we trust those people. So it's about creating trust, creating channels for people and communities and then putting them to use. This is one side of the creator, like, I own my likeness, I own my. What I say and how I being perceived. And then the other thing, the other side we are considering is the audience. Of course, the audience might want to see me in this podcast or might want to see someone else giving this podcast with another voice that he likes, with another face that fits him, or even with another language to be translated. And all of those traits, all of those modalities will be available in the future. You can consume the content at your pace, at your language, with, we call it, pleasant interactions. So we will be able, by digitizing people likeness and the way we communicate with each other, for content to be delivered through machines.

I'm picking up on a sort of, I guess a trend that you've been kind of alluding to, which is the fact that I think where this technology has been maybe misapplied is where it's sort of not accessible to a wider audience. And there's a sort of concentration of who can use it and who can't use it. But as you sort of make tooling, like the tooling that you're developing and give people sort of that don't have technical skills to like, spin up a GPU instance and like run TensorFlow, you know, in a distributed way across a cluster and all this stuff, like, as soon as you give, like a kind of wider audience the ability to create with these tools, they sort of create their own Persona and they have control over that. But if they're not able to like access that technology, then there's an imbalance of like who can use it and who, who can't, which kind of might produce some, some misusage. Because I'm thinking even of like audiobooks. It's like a non. Maybe this is a, this predates like the deep fake scenario. But like for a long time with audiobooks or with like things I was listening to on my, on my phone, I could switch a voice potentially like, oh, I want to hear, you know, or like Google Maps is a good example of this, right? Like I can change the voice that's going to talk to me from Google Maps. And that's like a preference thing on my end right now. I'm sure that there's some complicated technology behind it, but the control decision on like what is pleasant to me, like you were saying, is being made by me and maybe not by someone else. So how do you view the kind of, I guess the shift between, you know, what changes when the technology gets out of the hands of maybe people like our talking here that maybe know how to spin up a notebook and train a model and those that have no tech skills, but the technology is appealing to them. And on the creative side, what changes when we get this into the hands of that kind of audience?

I think it's incredible shift in creativity in the ability of people to communicate their ideas and basically to manifest what they know, what they think, either through text, through prompt, with reference images, everything that happening. And I think it's something that is happening with our industry, with the state of the tech, not only with synthetic media and virtual humans, but also with image generation and prompt invoked image generation? I think everything is very restricted now because the owners of technology is trying to. They know that there is risk and they're a little bit afraid of what might happen and they don't know. So it's growing and it's opening the community. I think of course I can't avoid stable diffusion and everything that's happening there. I think it's super interesting that it's going to be open. And I think in the end what you said about the ebooks is super interesting that people can listen to. I think we're coming from the angle that not only you can choose the voice, but you can also subscribe to have your voice read whatever books you want or willing to read. And then Chris might listen to those books in your Audio and you might be rewarded in some way for this. So this whole marketplace of skills and traits that we have is basically, I think one of the things that is being built and I think generally that technology is adapting and we find good uses and misuses. As I said before, I think the same we're experiencing with social media, with groups, moderation and stuff like this. So it's gradually expanding into our culture, changing, enriching our culture. And I think the future will be exciting.

Lior I, I think there's, there's a ton of things to explore in this, on this topic. Before we get too far, I would love to kind of give people an intuition for like what is possible with this technology and how. So the scenario I have in my mind is let's say sometimes Chris is out of town and I want to record a podcast with Chris, but he's not, he's not present. So I want to, I want to work with Chris to create a virtual Chris. And then when Chris is gone, I can just type replies to myself and then talk to virtual Chris back and forth, forth. And let's say Chris has given me permission to do this.

You have very low aspirations, Daniel. I just gotta say.

Could you explain Lior like what technology enables that? Like how would like from a technical side, what's needed to be put into place in order to materialize that scenario?

Yes. So we have language, written language, which is very easy and we have an easy way of inputting language and into the machines basically. And then we can take them and transform them into voice. With capturing the voice, there is voice cloning. A lot of stuff is happening in this field. There are many companies, great companies and other open source projects making this happen. With a few voice samples, we can have text to speech engine which basically creates the audio of Chris in that scenario. And then with other systems we basically can take the audio in whatever language that was generated by the text to speech and create those, this speech basically to the image of Chris speaking it in real time. If we are a vlog and basically we can create and the Phil is developing, but we can create types of looks for Chris, we can create more emotions or sentiment in his speech. And it's really, I see it as like we are in the early days, but basically on our platform and platform like ours, people can just jump in, write the text, choose characters, choose voices, choose languages, hit the create button and then as he said, invoke a few GPUs in the cloud and within minutes, if not seconds, you get a video or a stream of that actual experience happening. And I think it's, it's amazing and.

For the audience, so we can see each other, even though you're only here in the audio. And he saw the, the look on my face. So the, the question I was wanted to ask was like, I love the picture you're painting of what's possible going forward to get there. There's kind of, you know, going back to kind of the, the. The ideas of. Of trust and authenticity and stuff to get people to see the positive on that. Because it's very easy to see the scary sides of kind of that creativity, I think in most people's minds because that's been their first exposure to the field. But you've shown us that we can really take advantage of creativity and kind of optimizing situations. So how do you bring people along on that so that they understand that it's something worth engaging in? And I'll give you a brief analogy that's not directly in this field, but it's kind of close. Like the field that I'm in now. I know that we are moving into an age where automation can fly airplanes much better and much safer. And Daniel and I were just talking about this in our last episode. Then human pilots will be able to. And I say that as a human pilot also. But getting people to trust their lives quite literally into that and in the context of deepfakes, to trust that they can take that step and be part of that creative process, enrich their lives and see all the benefits. It seems like that will be a challenge to kind of bring people fully along the path in a broad sense, not just specific use cases, but in a broader thing. Any thoughts on how we navigate that kind of culturally and as humans together.

What we are trying to do is create positive use cases and positive. And just let people see the positive uses of such things. And actually in our company we see a lot of people that want to become characters and have their own character. And we communicate in work with our own characters with one on one sending Slack messages or videos and stuff like that, and having ourselves on the platform and be able to create that. And aside from that, we have a lot of people asking, when can I be my characters? Those early adopters. And I think as things play out and those use cases are out there and people see other people appear in content and they see it's safe and it's put to good use and are being rewarded for that, I think the general positivity is something that is built gradually and the trust is built gradually. And I think it's a good start that we've seen the misuses before, so we know what to watch out for. And going forward, we can start to see better and better uses of the technology of, you know, it started with the top, with Obama or Tom Cruise or Trump or whatever, examples of people frightening. And then we can just build from there with the good uses. And people are already in a place that might be frightened, but once they see positive uses, I think they might want to subscribe to this notion and join this basically this character economy or virtual human economy that is being built.

I like the idea of thinking about this like an economy. I was thinking in my own scenario, like, I have recorded videos for different, like, trainings and such around, like, AI and technical subjects in the past, which I love doing. But it is a lot of work to, like, get into an environment with the right lighting and arrange a person to record recorded and then produce the video and all of that stuff. And so there's been a lot of times where, like, I'll create what I think is a cool tutorial or something like that, but I'm sitting in an airport somewhere or like, you know, you know, just on my laptop. I can't record a nice video for that in an engaging way for. For an audience. But if I had this sort of virtual version of myself, I could see myself, you know, typing out text associated with that tutorial and pairing, you know, an engaging video with a screencast or something, which is much easier to produce right. On my. On my laptop in an airport. But then I was also thinking, you kind of brought up the idea of the economy. And it does seem like there is an incentive potentially for creators to do this because, well, what if I then had a group of people that, like, there was sort of a brand around the content that I was creating. But there were other people that had great tutorials and they wanted to maybe submit them to my trainings and put my face in front of it. And I liked their content, so I was getting more good content, right. But maybe I incentivize them financially for part of, like, the trainings that people subscribe to on my platform or whatever it is, right. And so there's like this flow of this nice kind of exchange of value between the two. As long as, like, I appreciate the content that they're doing and they understand that their face is not going to be on it, but maybe they're recognized in some way, right? And I, you know, understand that, like, I'm going to recognize them in some way, but my face is going to be on it. And that's part, kind of part of the brand. So all of this thinking is like, it's sort of flowing through my brain. Have we seen those sorts of creators finding these new ways to incentivize this yet? Or would you say we're still in a stage where people are exploring potential usage?

I guess, yeah, I think people are, from what we see, are in need for video content. They want rich content for their audience and then they're looking for ways to produce this content in an easy way. And I think a lot of people don't know about this technology or that it's even possible. Like just typing in your text, your narrative, building up the scenes. Like basically they know they can make a PowerPoint but they don't know they can make a rich video with a character in an environment. Just click a button and have a video play and put it there in, embed it, share it, upload it, whatever. So they're not there. I just wanted to say one thing because you made me think about something about your tutorial example. So things that you can record your. Basically what you're able to do is record your first tutorial and then get it into the system, transcribe into text and then you can keep it updated and change characters. And the other thing is not only that some people would like to consume this content with your face, some other people might want to consume this content, the same content in a different time or a different language with another character or another presenter. So those are the things that we are dealing with.

So to extend that idea out, would you predict that like the entertainment industry and actors and actresses and, and musicians and such will be out there offering their brands as a form of interfacing with an audience. And so you might have, and I'm obviously making this up, you might want to have like the movie Grease from way way back and Young John Travolta and Olivia Newton John are teaching.

Chris wants to be a Fonzie.

Chris, I'm telling you, this is pretty cool. I like this idea. I know it sounds a little silly, but bear with me. But you're able to basically select something that is, that has appealed to you. But you could then put content out there in that context and you could actually have brands extended into kind of user generated content where you have kind of deep fake brands supporting that. Is that I'm being a little bit silly for fun, but I'm not being too silly. Is that what you see in terms of this economy going forward?

Forward? Yeah.

I think like everything with economies there will be price fluctuations and of course famous people, A Listers and B Listers and everyone else will take part in this economy once it's grown and we've grown the trust and the ability to control where you're appearing and what are you appearing at what price or at what reward. I think it's already happening to today, like celebrities, Hollywood, A lister or stuff like that. They are advertising things in different countries that they might not advertise in their countries because it's other language and stuff like that. So. And I think if this technology can help them expand those reach and control where I appearing, at what prices, when I'm appearing, what is the content I'm delivering. And if we can build the structures to make these transactions flow, we can definitely make it work. I think I would participate. I'm now giving my content and my voice to this podcast, of course, and I might be able to participate with my voice in other places and give the content that I want to give to other places, not necessarily with my voice or my appearance or stuff like that. And then everything, the modalities of the content and the dimensions of the content will be basically just transactions and will be assimilated to be consumed by the viewership in the best manner. And this is that pleasant interaction that we mentioned before. Everything will be more programmatic and will be consistent, assumed at the right place in the right time, with the right delivery of the right content. This is what we think about.

So Lior, we've talked a lot about the sort of technology in general. Like, we've talked a lot about text, sort of natural language processing on the podcast we've talked also about like speech to text and text to speech sort of things. Recently I know we had Joshua Meyer on from Koki, they have great tooling around that. But this element of like once you have synthesized voice and then like pairing it with avatar that sort of has mouth movements that are matching up with the voice, that's something we haven't really explored from the technical side. Could you kind of touch us up on like what are the state of the art kind of models related to that sort of interaction? And like what sort of data do you need to have available to, to successfully do that sort of operation?

So we're gathering basically a video data of people speaking in different languages, different people with different appearances in different angles and stuff like that. And then we label the data with landmarks and with resolution. We align the data and we prepare everything else and then we. Basically what we do is we create a bridge or a latent space that basically can encode the audio and decode the face and then we can reconnect the audio in the back. And of course, we're using mainly guns and exploring different things. And in our field, the main interesting thing is video and its stability and temporal stability, things that is not required in other fields. For example, image generation now we see on different platforms with Dall E and stable diffusion and others, you have a seed basically, and a seed create the generation and then morphing between seeds is not always flowing. So we are dealing with a lot of temporal stability and correctness of the expression.

Interesting. Yeah. And I'm guessing that this sort of. It's very interesting to me, like how audio is represented within AI models and oftentimes like more like image than it is anything else in terms of like spectrograms and that sort of thing. But then like when I think of audio and language and video, you're going to have a certain sampling rate for that video, you're going to have a certain sampling rate for. For the audio. Those are likely going to be different. The sort of dimensionality of those things is going to be different and maybe even different between different samples. So I was wondering, just generally as you've kind of dug into this space, what are some of the, I guess the data challenges that you've experienced working with audio and video data in the AI space? And for those out there that are kind of digging into these newer models that are either processing audio or video or both, what recommendations could you make to people in terms of the challenges that you faced in kind of really digging into this topic?

Yeah. So I think to answer your question, the biggest challenge with data, as I see it, is data pipelines, once this data is captured, which is usually kind of easy, or all having the infrastructure to basically normalize, to clean, to align and to label this data and bring it into the gpu. So I think the infrastructure for doing it and updating it is super important for us. And aside from that, I think that clean data is of absolute like labeling and cleaning the data is utmost important. And for us, challenges that we might face is, for example, audio noises and stuff like that that are not necessarily. Or a different person speaking and not necessarily the person in the video that is captured or aligned. So those are the kind of things that we. Interesting. But as a general suggestion for all the listeners, I think thinking about the pipeline on how the end to end pipeline of how to acquire the data and then process it all the way from the camera or whatever you're gathering from a link on the Internet until it gets to the GPU through the data loader. We see it as one big challenge and trade offs along the way.

Yeah. And I guess it's likely that you like in terms of supporting the creation of an avatar for a specific person, for an actor in a studio, you might have a lot of control over, over a lot of that and be able to closely couple that. But as soon as you're passing things over the Internet, I'm sure that there's degradation and there's like all sorts of things that you could come across. So. Yeah, that's, that's super interesting. I'm wondering. So we've talked a lot about, about digital humans or virtual humans and this sort of avatar creation. You've kind of given a little bit of hints of like what's available and what you've built right now. I'm wondering if you could maybe summarize sort of like what the state of what you've built is right now and then maybe a couple things about what you're excited about, you know, looking into the next, next couple of years, what you think is possible with the sort of features that might be added there.

Yeah. First of all, we have our SaaS offering live in production. You can register, you can try our system. There's a free trial. You can create videos, you can select avatar, you can check out the technology with voices and everything and we have subscriptions model and you can continue and make videos on the go whenever you need them. Our focus is business use. We really think the world of work is a huge opportunity to create trust. And we believe that future generations are used to consume social media or social video and such and are expecting the world of work to change from text to be more rich and more engaging and, and more pleasant and interactive. And this is where we're going, this is where we're building. You can definitely sign up and check us out and can you repeat the second part of the question?

Yeah, I was wondering like kind of looking to the future and maybe Chris, you had something as well, but there's this sort of like text to avatar creation and the variability and the creativity that you can do with that. What are maybe some of those things that are on your mind and you're not committing to anything by any means, but like what are those things on your mind like? Well, if we could enable this in the product, that would Level it up a lot and you know, what are the things on your mind in that regard?

Yeah, definitely. So I won't expose everything that we gonna launch soon, but I'm just saying exciting things are on the way. We are super excited about prompt accessibility, image generation from prompt. And people are able to add media to those videos by the text or by the narrative. Recognizing this narrative and making prompt more accessible and prompt engineering, which is a big thing now in the industry of how you create the imagery that accompanies your narrative and create a compelling video in a rich environment. This is one thing we're very focused on, I think environments in general, 3D environments and rich videos and other things that create a whole experience, like basically watching tv. I think those experience will get closer there and we're super excited about more geeky stuff, which is like inversion, if you're familiar with that, that's embedding, referencing words into the prompt and then creating the. Using those objects or basically translating your likeness to another domain. For example, someone that looks like me, just with hair. You're not seeing me on the podcast, but I'm bald. So all those kind of things and ability to change something that looks like you, but have some of your traits. Yeah, sorry.

Someone who looks like me who is able to rollerblade successfully and.

Well, exactly.

I wonder this. I don't know if this is some of what you're getting at, but also kind of like I could see how some of that could be used to where you have an avatar in the voice, but you could also bring in, if you're talking about a car driving down a street and then this happens that's sort of generating this sort of almost like B roll for your video would seem quite interesting to me because there's so much of that video out there. And so in the same way that a lot of this text to image stuff is happening, it seems like you could generate some really compelling kind of transition shots or whatever that might be. I'm wondering, as we sort of get closer to the end here, one of the questions that our listeners might, might ask is, you know, from an expert in the field who is working in this every day, as you mentioned, this technology is only going to get more compelling, like TV quality, very, you know, high resolution, very compelling, you know, as some would write in the language space, like very coherent output, how would you recommend people think about like, how do I. It seems like we're getting into a space where I can't tell what's fake and what's real. You know, what would you tell people in terms of as I navigate the world and I look on social media and all that, is it even relevant anymore to think about telling the difference between those two or how would you recommend people kind of think about that certain aspect of kind of the cultural shift that this technology is causing?

I don't know how philosophical to get with this question but basically a lot of the discussions that we are having is of course Photoshop retouching and how people appear on social media and filters and all the stuff. It's a build up to this discussion. But we're thinking about AI at large and we like to think about not like what are we teaching AI and what is it learning? And we think about what AI is teaching us in some sense and then we think about what is he learning and then the bias in the models, it's actually a reflection of the culture. So we think basically it's trying to show us or to teach us in some sense what we are and then we can choose and build our culture with creating it's a two way communication from these new technologies to our culture. And I think it will definitely be exciting and as a culture we will decide where we moderate it in some sense.

Yeah, that's really, really good input. I think this is one of those conversations where the possibilities seem many and there's definitely going to be some things that like you say, you know, cultures, you know, governments, societies will have to, will have to wrestle with. But I think on the whole like I'm, I'm very excited to dive into some of these things. I'm really excited to jump over and, and create a few videos. I want to share a couple things in my own Slack channel and see, see what people's responses and, and if they recognize that this is, this is that generated video. But yeah, I really appreciate you taking time lior to join us. It's been an awesome conversation and looking forward to the amazing things that hour one is coming out with and hope to stay in contact and have you back on the show both in real life or as a virtual human, however you prefer.

Thank you for having me. It's been a pleasure talking to you guys.

Alright, that is our show for this week if you dig it. Don't forget to subscribe. Head to PracticalAI FM for all the ways and if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets to fly, IO for back in our dynamic requests to Breakmaster Cylinder for the Beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.