Emerging AI: This Week's Mind-Boggling Advances!

There's been a lot of cool AI and tech innovation that's come out over the last several weeks. And I want to break down some of the coolest stuff that I've come across over the last few days, starting with this really cool video that Mr. Boofy made of me in this same office here. He used a tool called warp diffusion, which is still something that's beyond me, but I kind of want to learn how to use it. Once I figure out how to do this myself, I might make a tutorial on it, but it's a little bit more complex than most of the tutorials I've made to date. But I just thought this was a really, really amazing video. So I wanted it at the beginning of this video. Shout out to Mr. Boofy for creating this. Everybody should go follow him on Twitter. He creates all sorts of cool videos in this same style here. I'm gonna start off with some cool research and stuff that's in the works, and then I'm gonna wrap it up with some cool tools that you can use right now, today, starting with this research called Mind Video. Now, in the past, I've shown you research where they're scanning people's brains with MRIs while showing them images and then using a process to actually convert what the MRIs are seeing back into images, essentially allowing us to read people's and see the images that they're seeing. Well, now they're starting to be able to do that with video. You could read about this latest research over@mind-video.com, but basically what they're doing is they're putting people into an MRI machine. They're showing them videos while they're in the MRI machine, using their brain encoding process and then decoding what the brain is seeing. Here's an example video. These are frames from the video of a cat here. And then what was reconstructed with was this video from the brain. Now, they say what this is doing is semantically reconstructing the video, meaning that it's getting the essence of what they're seeing, it's getting the idea, but it's not showing exactly what they're seeing. So here's some other examples. In this one, the video on the left was a video of some fish that they were showing them. The video on the right is what the brain scans produced, which you could tell is kind of an underwater fish sort of video. Here's another one where the one on the left shows a video of like a jellyfish, and on the right, it's a fish. You can Tell it's underwater in the ocean. It sort of got the idea. Here's another one. They showed him a video of some horses. You can see on the right, some horses on the left, a turtle. It translated more into like fish, but it's still the underwater animal scene. They showed him this video on the left of a man walking with his dog. I believe what their brain decoding, decoded was this video here where it looks more like a couple walking with a dog, but kind of going in a different direction. Here's a video clip of a man and woman side by side. And this sort of semantic response was this where, where it's a man talking to a woman. Here's another one with a scene of a car driving through a little toll booth and then a river and then the result was sort of a mountain road. We got a little video of a bird here. What it saw was this bird and then finally a video of a crowd walking. And on the right, a video of a crowd walking again. The past research was mostly doing image to image where they were showing them images, scanning their brain and then getting similar images back. Now they're starting to be capable of doing this with video. It's not perfect. It's still semantically correct. It's getting the sort of essence, the idea of the video that they're seeing, but it's still not perfect. But this is brand new research. This is very, very early, so exciting to see where this is going again. I don't really think we have to worry about, you know, governments using this technology to read our minds or anything yet, because it still requires people being put inside of an MRI machine with their brains being scanned. And I don't believe that governments are anywhere close to being able to do this at a large scale on a lot of people. So I think our thoughts are still safe for right now. Now here's an piece of tech that I came across on Twitter from Bjorn Karaman. I'm sorry if I mispronounced your name, but this is basically a camera with no lens. Here, take a peek. So he adjusts some dials on the camera. You can see he holds it up, he adjusts the meters that it's taking the picture of and he snaps a photo and it analyzes the scene and then essentially writes a prompt and then generates an image based on the prompt. So that's the original scene there and that's the image that it took based on creating a prompt and then generating the image off the prompt. Now, while I think this technology is really interesting and cool to see. I'm not 100% sure I understand the practical use of it, especially if I could get on like Google Maps, find a location and get a similar result. And if I'm actually in a specific location and I have a camera with me, I'd probably want to take a picture of what I'm actually seeing. But let's be honest, it is a pretty interesting and original idea. I'd love to hear what people think might be some interesting use cases for technology like this. So earlier this week on Sunday, Jensen Huang, the CEO of Nvidia, gave his first keynote presentation in four years. And I'm going to break down more of that keynote presentation in Friday's News Breakdown video. But one thing that he showed off that I thought was really cool was this video where a real human goes and interacts with somebody inside of a video game. And the person in the video game responds real time and their response is generated by AI. It's not pre written dialogue. So this video I do have shared on my Twitter. If you want to watch the full thing here, make sure you're following me over at Mr. EO. But let's take a peek at this video here.

Hey Jen, how are you?

Unfortunately, not so good.

How come?

I am worried about the crime around here. It's gotten bad lately. My ramen shop got caught in the crossfire.

Can I help?

If you want to do something about this. I have heard rumors that the powerful crime lord Kuman Aoki is causing all sorts of chaos in the city. He may be the root of this violence.

I'll talk to him. Where can I find him?

I have heard he hangs out in the underground fight clubs on the city's east side. Try there.

Okay, I'll go.

Be careful, Kai.

None of that conversation was scripted. We gave this Gen AI character a backstory. His story about his ramen shop and the story of this game. All you have to do is go up and talk to this character. And because this character has been infused with artificial intelligence and large language models, it can interact with you, understand your meaning in a really reasonable way.

So like Jensen said, that was all generated with AI. I would have loved to see them have a different conversation because the conversation they had in that video looked like it could have been pre scripted. Right. He walked into somebody's ramen shop. Asked how he was doing, the guy said, my ramen shop's been caught in the crossfire. Guy asked, well, who do you think did it? He gave him some information and it felt like it could have been pre written dialogue. It would have been really interesting for the end user to walk into the ramen shop and have a little bit more of an in depth discussion before diving into what he needed to do next because otherwise it's sort of not really too different from having pre written dialogue. Now one thing that's really cool is one of the creators of this technology, the founder of Convey, reached out to me and we're gonna have a chat and he's gonna give me a little bit deeper of a walkthrough of how this technology actually works. So expect a follow up video where we dive even deeper on how this technology works and where it's headed. Now this idea of being able to have conversations with NPCs and video games and have the conversation be generated by AI isn't super new. In fact, my buddy Matt Vid Pro here did a video about it two months ago where he streamed playing an AI game. The big difference is the timing, it's the latency. Right when we just watched that in video demo, the person asked a question and the response came nearly immediately. Watch how the existing games that are on the market that do this kind of thing work. Right now just pay attention to the pause in there.

Listen, what's your name? Can I call you sweetheart? If you don't mind. Do you have a family, do you have kids? And most importantly, do you play Among Us?

I appreciate your attempts at small talk, but let's keep our focus on this investigation. My name is Sergeant Shannon and as for your questions, I am not comfortable sharing personal information with you. And as for Among Us, I'm afraid I haven't had a chance to play it yet.

What are you doing this week?

Weekend?

I gotta ask.

Detective, as much as I'd love to chat about weekend plans, we have more pressing matters at hand. Let's stick to discussing relevant details about this case and work together to ensure public safety.

So you can see there's quite a bit of delay between Matt asking the question and the response. This video is actually hilarious. Matt asks a lot of really funny questions so I'll make sure it's linked up below in the comments so you could watch his full live stream and skip around. Some of the stuff he tries to get the AI to say is pretty hilarious, but I won't spoil it. You should check out his video. Continuing on with the them gaming. This is brand new research called Voyager that's actually training the computer to play Minecraft. It could do things like mine, Amethyst, Gather Cactus Build a base, fight enderman, build a house, mine gold, build nether portal, hunt pigs, swim, and basically do all of the tasks that somebody might need to do in Minecraft. Now, they obviously don't know that rule number one of Minecraft is that you never dig straight down.

Never dig straight down.

But this is a learning computer. Now, the idea of a bot that learns to play Minecraft, on the surface may not sound that impressive, but the technology and what they built underneath is actually quite cool. So basically what this model is trying to do is it's trying to become the best Minecraft player in the world. And the way it's doing that is it's by going and learning how to do different skills. And once it figures out how to do a skill, it saves that skill into its own internal memory so that it can remember how to do that again later and then moves on to learning another skill. And then it saves that skill as code inside of its internal memory. And it repeats this process and repeats this process until it gets better and better and better at Minecraft. So it's a learning bot that's essentially teaching itself to get better at Minecraft once it gets good at one skill, saves that skill inside of its code memory, and then continues to stack skills on top of it. While the use case that they're demonstrating this with is Minecraft, the underlying finding technology of this is really, really powerful. When you think about it is a self learning bot that will learn a new skill, save that skill to memory, move on, learn the next skill, and continue to stack skills on top of itself until it's the best at the final larger skill that is trying to accomplish. So here's why they specifically chose Minecraft. Unlike most other games studied in AI, Minecraft does not impose a predefined end goal or fixed storyline, but rather provides a unique playground with endless possibilities. You're not trying to necessarily beat the game, you're just trying to get better and better and better and explore and build and just figure out new tasks to do in the game. An effective lifelong learning agent should have similar capabilities as human players, so it will propose suitable tasks based on its current skill level and world state. Learn to harvest sand and cactus before iron. If it finds itself in a desert rather than a forest, it will then refine skills based on environment feedback and commit mastered skills to memory memory for feature reuse in similar situations. For example, fighting zombies is similar to fighting spiders and continually explore the world and seek out new tasks in a self driven manner. And it's sort of this perpetuating cycle that will continue to loop, seek out new tasks, learn how to do those tasks, commit them to memory, continue to seek out new tasks, learn how to do those, commit them to memory, repeat the process until it becomes the best Minecraft player in the world. When you really understand what they're trying to accomplish and you understand that it's not just about Minecraft, you can see how this can be powerful in other areas. It's essentially an autonomous agent, but created for use in a Minecraft world. And to top it all off, they've actually open sourced the code and put it on GitHub so you can install it on your own computer and train your own Minecraft agent to get better and better if you'd like. So here's something else that's in the works that was sent to me by the people over at Invideo to check out. Now, we've seen a lot, lot of text of video already where it will generate short clips where they're, you know, maybe three, five seconds, all the way up to 15 seconds with stuff like Gen 1, but even that requires a video input. Well, what Invideo is doing is creating longer form text to video where you can type in a prompt where you expect a longer video. And they did send over this demo video. So let's take a peek at it. A cinematic video about a monk with superpowers. It's generating music, generating sound effects like, and this is the video it generated. I'm not playing the music because I don't actually know the licensing on it, but you can see there's this video of a monk, another monk, meditating. Now I don't believe this is AI generated video. I think maybe what it's doing is it's creating the narrative for the video and then finding stock video to go along with it. So I don't believe what's actually being generated is AI generated. It's just being AI sourced video, so to speak. But, but the people at it in video, I am going to have a conversation with them soon and they're going to clarify this for me and we'll be demoing this more. But we're getting to a point now where you can enter a prompt for a full longer video and it will write the script, find the music, generate sound effects, pull in all the video clips that you need and generate that video. Now it'll get really exciting when it starts actually generating that video purely with AI. But right now it's still pretty damn cool that it's going to be able to generate it using stock footage that it finds from around the Internet to go along with the narrative that's on the screen. So I'm excited to see this either way. All right, now I'm going to move into some of the stuff that you can actually play with right now if you want to. Starting with this cool video that I came across for a tool called ecout, which I believe is the French word for listen. It's a tool for online interviews. If somebody's interviewing you for a job, it will actually listen to the questions that they're asking during that job interview and then feed you the best response that you can give for that interview question. Now, this isn't a real interview here. This is sort of a demo interview. But check this out.

What about your development areas? What do you have identified as your greatest and biggest improvement areas?

You can see over on the right, you can see what the speaker is asking. Even to the right of that, you can see the recommended response to the question.

And what have you done to improve them so far?

I would say my greatest development area is my communication skills. I work on improving my ability to clearly convey my thoughts and ideas to others. I've been working on improving by practicing my active listening.

And so you can see that as the interviewer asks the questions, it listens to the interviewer and then gives you the best response to that interview question. Which is kind of funny, because doesn't that sort of put everybody on a level playing field when it comes to doing an online job interview? As this sort of technology gets more prolific, I really doubt companies are going to be able to do interviews in this fashion, because pretty much anybody can be eloquent and well thought out with their responses to interview questions. Again, this is open source. You can actually install it if you're on Windows. It says it's only been tested on Windows so far. Doesn't mean you can't use it on a Mac. They just haven't tested it on a Mac yet. And all you got to do is follow these instructions to get it set up. I'll make sure this GitHub is linked below the video in the description. So if you want to check this out and play around with this tool yourself, it will be linked up below. All right, so this is one where they reached out to me and sent me this demo, and I was blown away by it. So we've seen tools like DID right, where you can upload an audio file and then upload a picture, and it'll make that Picture, sort of talk like the audio file. I've done a handful of videos demonstrating it and have all sorts of use cases for it. Imagine that. But instead of starting with a picture, you're starting with a video. So, for example, here's a video from Harry Potter of Voldemort walking forward and then looking into the camera. And then let's say I want this Borat voice on the video.

He is my neighbor, Nursultan Tuliakbai.

I could click generate lip sync. It's gonna process this. It's fairly quick to process.

He is my neighbor, Nursuktan Tuliakbai. He is painting my assholes. I get a window from a glass, he must get a window from a glass. I get a step, he must get the step. I get a clock radio he cannot afford.

Great success. So you can see whoever's on the screen in that moment, it actually moves their lip and lip syncs it to the audio that you uploaded. And you can upload your own videos and you can upload your own audio file and do it with any clip. So, for example, let's go to 11 labs real quick. I'm going to use my cloned voice here. I'm going to say, my name is Matt Wolf and reporting for duty, sir. Let's generate that.

My name is Matt Wolf and I'm reporting for duty, sir.

Let's go ahead and download this clip that we just made. I've got this clip from Forrest Gump. Let's add our own audio. I'll click upload audio and grab this 11 labs audio I just made and let's generate lip sync.

My name is Matt Wolfe and I'm reporting for duty, sir.

And now I just made my voice over Forrest Gump and his lips moved to what I just said.

My name is Matt Wolf and I'm reporting for duty, sir. My name is Matt Wolf and I'm reporting for D. My name is Matt Wolf and I'm reporting for duty, sir.

So did is great for animating still images, but this Lalamu tool, this is what we're going to start using to lip sync over existing videos. So my prediction, if you're on Twitter over the next several months, you're probably going to see a lot of these types of videos circulating pretty soon. All right, finally, I want to wrap up with talking about Adobe Photoshop's Generative fill just one more time. I know I've talked about it in a couple different videos before. I made a full video about it the other day, showing off some of My own experiments with it, plus some stuff other people did. And some amazing commenters on this post shared some tips to actually use generative fill. Even better. For instance, don't use the magic wand to select the generative fill area because you want a bit of an overlap with the original image. I think when you use generative fill, you have to make a selection where it's slightly overlapping the original pic. So that same comment was shared by a handful of people. So I just wanted to touch back into Adobe Photoshop's Generative fill one last time and try a couple more quick tips that I've seen with Generative fill. So if you remember, I was playing around with this image the other day and I tried to add some legs to him, but instead it put a countertop. So I'm going to zoom out a little bit. I'm going to go to my crop tool and I'm going to go ahead and extend this down quite a bit here. Now I've got a whole bunch of extra area below his body. Let me zoom out a little bit more just so we can see the whole thing on screen. And I'm going to go ahead and grab the selection tool this time, select this whole area, but then make sure it overlaps a little bit with what's above it. And now let's try the generative fill. And would you look at that? It actually gave him crazy monster hands, but it did give him legs. So let's see some of the other options it gave. There's that one looks much better with a closed fist for sure. And then there's that one. So pretty dang impressive. Now, another cool trick that I've seen was taking two different images and then using generative fill to fill in the gap for those images. So let's go ahead and create a new image here. And I'm going to pull in two separate images. I'm going to pull in this image that's supposed to be me, like as a wizard. I'm going to make it a little bit smaller. Let's put it down. Actually, no, let's put it right here, right here. And then I'll pull in this image over here that's got these two wolves in it. They look like they could be sort of similar. So what I'm going to do is I'm going to merge these two images together so that it's using both of them for context. Let's merge the layers. Now, it's just one layer here. I'm going to use my magic wand tool to select the background, but then I'm going to bring it in a little bit so that it overlaps with the images a tiny bit. So if I go to select and then modify and then expand. Let's expand it by 10 pixels and you can see now it overlaps into the images a little bit. And let's just go ahead and do Generative and see if it figures out how to connect these two images together. So there's one. You can definitely see a hard seam down here where the two images kind of split, but it gave me more body. It kind of tried to combine the trees. Let's see another alternative. This one's not too bad either. Less of a seam. It kind of blurred the lines there. And then this one's pretty good as well. The color palettes are slightly different, so you could tell that it started as two different images. This is a little bit more hazy in the background. That, to me, is really, really cool. This image was generated in stable diffusion. This image, I believe, was made in midjourney. I pulled both those images in use generative fill, and it kind of made them all look like one image. Now that's pretty mind blowing to me. One last little fun thing I've seen people doing as a meme on Twitter that I want to try myself is to expand an album cover. So let's go ahead and create another new image here. I grabbed this iconic album cover from, you know, my high school college years. So let's go ahead and add that in there. I'm going to use the Magic Wand tool again to select and then I'm going to modify the selection and I'm going to expand it a little bit, 25 pixels so that it's seeing a little bit more of what's going on in the image here. And then I'm going to use Generative fill to have it generate what it thinks is the rest of the album cover. There's one iteration of it. Here's another. And here's another one. Can't really tell what's going on in really any of them. Let's go ahead and generate again and see if it comes up with something even funnier. Here's one where she's in the water. I don't know what's going on with that one. I don't know what's going on behind it, but it looks cool nonetheless. Wanted to make that little update to Photoshop Generative fill. Try some of the stuff that you guys were suggesting in the comments. Try this meme generation that I'VE been seeing on Twitter and have a little bit more fun with it because Generative fill is really impressive and I think in my last video where I talked about it I kind of undersold it a little bit. It is really fun and really cool and that's all I got for you today. I have a couple requests of you if you enjoyed this video. Follow me over on Twitter I am MrEflow on Twitter. I share AI news almost as it happens. I share some of the coolest AI tools that I come across as I come across them. I make a lot of extra videos that are only available on Twitter that I don't share on YouTube. I like to make super edits of longer videos where I take two, three hour long videos and edit them down into like 10, 12, 15 minute videos that are just the tldr of the video. I share all sorts of cool stuff on Twitter that doesn't make it to my YouTube channel. So if you're not following me on Twitter, follow me over rflow. And as always, if you like nerding out about all of this AI stuff and you want to find all the latest, greatest tools, you want to keep in touch with all the latest AI news. In addition to my Twitter Twitter, I also keep this Future Tools website up to date on a daily basis. If you just want a once a week breakdown of what's going on in the world of AI, join the free newsletter and every Friday I'll send you the TLDR with just my five favorite tools, a handful of news articles, a handful of YouTube videos, and one cool way to make money with AI. All you gotta do is go to FutureTools IO and click Join the free newsletter and I'll start hooking you up this Friday. And final thing, the last call to action. I know if you're in the marketing world, this is way too many things to ask of a viewer. But maybe give this video a thumbs up if you liked it and subscribe to the this channel if you haven't already. I really really appreciate it. Thanks for nerding out with me. See you guys in the next one.

Bye bye.