AI News: Autonomous Agents & New AI Video Tools

This week has been one of the craziest weeks in the world of AI in a long time. And I know I sound like a broken record, like I say that every week, but this week there was legitimately a ton of news coming from places like Anthropic OpenAI, a whole bunch of image generators, a whole bunch of video generators, new robots, all sorts of really, really cool stuff. I don't want to waste your time. Let's just get straight into it. I'll do my best to sort of rapidly fly through all this news to make sure that you're fully looped in on all of the biggest events happening in the world of AI right now, starting with biggest news of the week, in my opinion, coming out of Anthropics Claude. And this week they gave us the ability to actually let Claude take over our computer and actually use tools on our computer. Now, I'm not going to go super deep into this because I did an entire breakdown as well as step by step tutorial on how to set this up yourself. So if you want a really deep dive, make sure you check out the video that looks like this. But here's a quick example they gave. In one of their demo videos, they have this spreadsheet here as well as this vendor form here. And they give it the prompt, please fill out the vendor request form for Ant Equipment Co. Using data from either the vendor spreadsheet or search portal tabs in window 1. List and verify each field as you complete the form. In Window 2, it takes screenshots of whatever's going on on the desktop to sort of acclimate itself with where it's at and what's going on at the screen at any moment. And then based on the screenshot, it will take an action. You can see here, it's searching the vendor database, found some info on their vendor database, and then automatically started pulling in data to this form. The right side. So again, what it's doing is you give it a command, it screenshots your desktop so it knows where to move the mouse or what step needs to be done. It does that step screenshots your desktop again, to confirm that that step was done and the right actions were happening, and also to figure out what the next step is. And it keeps on repeating that process of doing a task, taking a screenshot, doing a task, taking a screenshot, until it completes the action that you initially prompted it to take. Again, I did an entire breakdown with a step by step tutorial on how to set this up yourself. So make sure you Check out that video if you haven't already. But that wasn't the only news that came out of Anthropic this week. There was also a new Claude 3.5 sonnet model and a new Claude 3.5 haiku model. We can see in the various benchmarks here that it outperforms the old versions of Claude 3.5 sonnet and in most cases the new sonnet even outperforms GPT4O. But anthropic wasn't done this week. They wanted to improve their platform even more, so they rolled out the analysis tool over in claude. Here's an example they gave on their website. Please visualize the sales funnel progression from awareness to purchase with a bar graph so I can identify conversion bottlenecks and they upload a CSV file of sales funnel data. It then creates some JavaScript to do the analysis as well as to create the visualization over here. If you have a cloud account already, in order to turn this feature on, simply come down to your name in the bottom corner. Click on this little feature preview button here and we can see the analysis tool is currently turn. We can go ahead and flip that one on. We also have the latex rendering turned off which allows CLAUDE to render mathematical equations. Let's go ahead and turn that one on as well and then we can close out of this and we now have that functionality. I just created some dummy data over on this side called mocaroo.com we could take a look. It is a CSV file here with a name, an email, a gender and an IP address. So we'll just go ahead and use this data as a starting point so that we can do a quick test here. So I'll go to Claude, pull in our CSV V file and we'll tell it to analyze this data and create a pie chart based on gender. I'm hoping this uses the new JavaScript feature to do it. So let's go ahead and find out. Yep. So we could see it is now analyzing this data, writing some code for the analysis. Now that we have the data, I'll create the pie chart. So it is now writing code for our pie chart here and once it's finished processing we can see our graph here with 48.8% female, 42.4% male. Pretty cool. All I did was give it this one short prompt and this CSV file and it processed it all for me and gave me a really nice looking visual. This is obviously just scratching the surface of what you could do with this new JavaScript analytics feature that's built in, but I wanted to do a quick test on video to see how it worked for myself. So as Anthropic inches closer and closer to having true AI agents that can do our work for us, Microsoft claims that they are now unlocking autonomous agent capabilities inside of Microsoft Copilot Studio now this article here on Microsoft. There's kind of a lot of fluff in it, but if I scroll down here, we can see the new capabilities that are coming to Copilot Studio. Things like Autonomous Triggers Agents can automatically respond to signals across your business and initiate tasks. They can be configured to react to events or triggers without human input that instead originate from various tools, systems or databases, or even scheduled to run hourly, daily, weekly or monthly. Each business process could have different paths, since agents create dynamic plans on the fly to handle and complete tasks. Users can then view the underlying logic for each of the agent's paths, which include key details, steps and systems involved. It's using the newest models like OpenAI's Zero1 model to help run the autonomous agent, and it looks like they're going to be actually demoing them during the Microsoft Ignite event that is coming up. Meta this week showed off a whole bunch of new AI research, including segment anything 2.1, Spirit LM, layerskip, salsa, lingua, open materials 2024, mexma, and self taught Evaluator. Now I'm not going to get too into the weeds on each one of these, but I do want to hone in on this Spirit LM here. Spirit LM is a language model that can take both text and audio as input and give both text and audio back as output. So some of the examples that they show down here on their demo page is they give it a text prompt 123-45 and the language model responds with.

An audio prompt 6 7, 8, 9.

Or here it starts with an audio.

Prompt A, B, C, D, E and.

We can see that it gave back a text prompt F, G, H, I, J, K, L, Monopede, etc. They gave it a text prompt. The largest country in the world is Russia.

Our country has about 150 million inhabitants.

And here's another audio prompt.

Yellowstone national park is an American national park located in the and you can.

See it went on to continue it in written text. They do say up here that these samples are cherry picked samples. Not everything we see out of this model is going to be as good as what we're hearing and seeing right now. Now these were from the Spirit LM Base model, but they also have the Spirit LM expressive model. So the prompt here. I am absolutely thrilled to be embarking on this new journey. It's going to be an incredible adventure.

And I'm so excited to have you guys welcome with me. The edge here for 12 weeks and there's going to be tons of stuff, incredible guests on the show, but I also just wanted to go.

That's what we get back. I mean the audio quality is kind of not very good, but you can hear it's got that same like fast paced excitement energy to it. Here's some initial audio prompts.

No, I think you're thinking about Days of Thunder, which is a Tom Cruise racing movie.

And then it goes on to give a text response here. So that's some pretty interesting research out of Meta. I'm not 100% sure what the actual use case is for it, but really I'm just trying to keep up with the latest research and what's going on and I'm sure people in the comments will let me know plenty of ideas and ways that this new research could be used once it's actually available to us to use it. Meta also introduced their quantized Llama models, which are much smaller models designed to run on mobile devices. Now, I'm going to really oversimplify this, but essentially what a quantized model is, it's kind of like compressing the model. It's sort of like taking this large model with tons of data and pulling out some of the data that it finds as less necessary to shrink down the overall size of the model. Now that's sort of a bad example. One of the analogies I heard was imagine you have a box of crayons and it's a giant box of crayons and there's 40 shades of blue in that box of crayons. Well, a quantized model would be taking just a percentage of those crayons and putting them in a new smaller box. And now Instead of maybe 40 colors of blue, you only have 10 colors of blue. Now blue's still in there. You could still draw and get blue into the pictures that you color. You just don't have as wide of a variety of shades of blue in it. Right? So that's kind of the analogy for quantization. You're removing a lot of sort of redundant data that ideally, without it being there, the language model can still kind of get to the same end result. Well, Llama just did that with some of their models in order to help them better run on smaller Devices like mobile phones. This video is sponsored by Opus Clip. Now, if you're not familiar with Opus Clip is it is this really amazing repurposing tool where you can take long form videos. It will find the clips that have the most potential to go viral and convert them into shorts with templates and text and captions and the whole works. They also just released this new Clip Anything feature which allows it to find any specific moment from your videos to clip. It actually skims the whole video and uses AI to sort of tag all of the things that are going on in the video and then all of those elements become searchable. So people, objects, actions, emotions, all of that sort of thing is now searchable inside of your videos and you can have it generate. A short form clip has the potential to go viral. When it creates these little clips, it actually gives them each a score based on the hook, the flow, the value and the trend. This top clip that it found here is 57 seconds long. It's a 99 out of 100, meaning that it's a clip that's very likely to do really, really well on places like YouTube shorts and Instagram reels. So check out how easy this is here. I can drop a zoom link, a twitch link, a LinkedIn link, a YouTube link, whatever I want, or I can upload the video directly. I'm going to grab the link here from my latest YouTube video and copy it. We'll jump over to Opus Clip, paste it right here. Down here I can select the clip length. I'm going to go 30 to 59 seconds. I like keeping them under a minute. We'll go ahead and let AI detect the style of content and then we can give it any prompt we want. A video did include news about the latest SpaceX launch. So let's go ahead and use the prompt, find portions that include a rocket ship and get clips. And after a few minutes of processing we could see that it found a clip from my recent video there. I showed off a rocket ship. So it found that rocket ship. But these subtitles on the video with like little finger arrows and stuff and it says it has a pretty dang high score of doing well. Other thing that I think is really cool about what Opus Clip does, this was originally a horizontal video where I was off in the right corner of the video and I was showing something on my screen. Notice how it reframed it and put my head down on the bottom of the video and then what I was showing on the screen at the top did all that reframing automatically. That is really cool and really handy. Check it out at Opus Pro Clipanything and I'll put the link in the description. Thank you so much to Opus Clip for sponsoring this video. IBM, who I am planning on making some more videos about in the near future, just released some new large language models called Granite 3. IBM said the new Granite models are designed as enterprise workhorses for tasks such as retrieval, augmented generation, classification, summarization, agent training, entity extraction and tool use. They can be trained with enterprise data to deliver the task specific performance of much larger models at up to 60 times lower cost. So they really seem to be designing their models for internal enterprise where you can train it specifically on your enterprise's data and get exactly what you need out of it a lot faster and cheaper than if you were using like Google's models. They also released this under the Apache license, meaning that others can use these models and build off them and iterate off of them. Xai, the creator of the Grok large language model, just launched an API, so now developers could go and use the GROQ language model inside of their programs. So we could probably expect to see all of the tools that integrate with all the large language models to also now include Grok as one of them. And we can also probably expect to see some tools that are a little off the wall and uncensored because Grok is pretty dang uncensored. We did get a little bit of news out of OpenAI this week. If you're a Plus user in the EU, Switzerland, Iceland, Norway and Liechtenstein, they now all have access to the new advanced voice mode. But even bigger news out of OpenAI this week came from somebody who just left OpenAI. OpenAI's senior advisor for AGI, Miles Brundage, is leaving the company and he wrote up a blog post as he was leaving, basically saying he doesn't think the world is ready for AGI. In his blog post he also made the comment that he doesn't believe that what the research labs have access to is a whole heck of a lot better than what the general public has access to. So he's basically saying that we're seeing the best that OpenAI has available right now. And yes, maybe they're keeping some other models under wraps, but the gap between what we have now and what they're keeping under wraps is not that big of a gap. Okay, let's talk about AI video because there's been a lot of announcements in this space this week, starting with runways announcement of aka Act 1. Now this is something we don't have access to. But they do claim they're rolling out right now. I've been refreshing my Runway account like crazy to see if I have access. I still don't have it. So it's definitely not fully rolled out yet. But basically what this act one can do is it can take the facial expressions, the emotions and the words that are coming out of your mouth and sync it up with an animated character. So check out this little 22nd demo here.

So let me get this straight. You came all the way down to the Department of Motor Vehicles and didn't bring your driver's license? Do I understand that correctly?

And here's some more demos that I'm actually playing on mute because there's music and I don't know the copyright status of the music. But we can actually see that these are just cartoon characters that the lips are synced, the emotions are synced. So as they move their head, as they show excitement, as they show anger or fear or sadness or whatever emotion that expression comes across on the cartoon character, and the lips are synced to the person talking. This is really, really cool. Now Runways Act 1 page, which I will link up in the description, has all sorts of demos of what this is capable of. So you can take a peek at some of these other examples here and see what it's capable of. But hopefully, fingers crossed, we have it in our hands pretty soon and we'll get to play with it and make our own videos from it. Moving on to video generators we do have access to right now. Let's talk about genmo and Mochi One. Now, Mochi One is another open source video generator. Since it's open source, if you have a strong enough GPU on your computer, you can actually download the files and run this locally. Also, since it's open source, people are going to tweak with it and iterate off of it and mess with it. And we're probably going to see some pretty uncensored video models in the near future. Whether that's good or bad or not, that's yet to be determined. But we're probably going to see some pretty uncensored stuff as more and more of these open source video models become available. Now, if you do want to play with this Mochi One yourself, you can. The site Fall AI has it available. This is the same site that I showed in previous demonstrations on how to set up Flux 1. Now there is a slight cost, so I have some credits here in my Fall AI account. But if we go to Explore. We can click on text to video here and we can see Mochi1 is one of the options here. Once inside, we have the option to enter a prompt, give it a random seed and enable prompt expansion, which I'm guessing just sort of improves whatever prompt you plug in. I did my normal test of a wolf howling at the moon and this was the video that it generated. It kind of looks like a wolf sort of merging and then walking towards the camera. It's not bad, but I wouldn't put it at the same level of quality that we've seen from some of the other video generators yet. It was really, really fast though. I believe it under a minute to generate this and we can see that it cost 40 cents per video. So this little clip here, because I ran it on fall AI and it's using their servers, they charged me 40 cents here. I went and tested it with my other favorite prompt, A monkey on roller skates. Once again, cost me 40 cents to generate this. But here's our end result. We definitely see a monkey. We can definitely see roller skates. Whether that's a monkey on roller skates or a monkey sort of playing with a roller skate. Yeah, I don't know. But that's what it can generate. Next up we have Hyper 2.0. This is another AI video generation model. We can see their demo here over on their X account. And I mean most likely these are going to be cherry picked videos. If you're making a sort of demo reel or a commercial for what your video player is capable of, probably going to cherry pick some of the best results. But these are some of the videos that have come out of Hyper H A I P. They all look pretty good, but again probably pretty cherry picked. You can actually go and generate videos with this one for free right now. If you head on over to Hyper AI again, that's H A I P E R A I they give you 300 credits when you first start. Now I don't believe that refreshes every month or every day or anything like that. I think you just get 300 credits to go and play with the video model. Once your 300 credits are up, you either can't use it anymore or you got to buy more credits. But to generate a video it costs roughly 30 credits. So you can get about, about 10 videos as you play with this. Before I show you some of my results, here's some of the demos that they have on their front page here of people doing various dances. They all look pretty decent. Not the best AI video we've ever seen, but they seem to be pretty good for generating these little, like, dance videos here. They're pretty good looking. I'm pretty impressed with what this can do, but again, fairly on par with most of the other video models we've seen so far. They do have both text to video and image to video in hyper here. And here's what I managed to generate. I did my Wolf Howling at the Moon prompt here, and when I blow this one up, you can see this one looks a little bit more like an animation of a wolf howling at the moon than the last one that I got out of Mochi. The moon's moving a little bit more than I would like it to, but other than that, the wolf looks pretty good. I asked it to generate a monkey on roller skates for me and you know, to be honest, this is one of the better videos of a monkey on roller skates that I've gotten. Now you can't actually see this sk, but you can tell the people around him are skating and he's got something strapped to his feet. I think you could tell it's a monkey roller skating. I don't know what this weird artifact is that appears and disappears over on the left. That's kind of funky. Not bad at all, honestly. All right, if you thought a lot was happening in the world of AI video this week, wait till you see all the stuff that happened in the world of AI image generation this week, because that area is even crazier. Let's start with the fact that we got stable diffusion 3.2. Ever since black Forest Labs came out with Flux Stability, AI has been kind of quiet. We haven't really seen a whole lot come out of them. We did get that previous model of Stable diffusion, which sort of went viral for all the wrong reasons. It was generating bodies that were just like legs, a torso and then legs, or like three heads, or they were just sort of like this nightmare fuel. It looks like they've sorted some of that out because this image right here I think is showing that you can get it to prompt a woman lying in the grass, which before it absolutely could not. And Stable Diffusion is open source. It says it's free for both commercial and non commercial use, and you can run it on consumer hardware. So you could Download stable diffusion 3.5 and run it on your own computer using something like comfy UI or possibly automatic 1111. There were two models that were just released with Stable diffusion. Stable diffusion 3.5 large with 8 billion parameters, and stable diffusion 3.5 large turbo, which is a distilled version of Stable Diffus stable diffusion 3.5, which is considerably faster than the large model. But since this large model generates Images At a 1 megapixel resolution, you're definitely going to get better quality out of the large model than the Turbo. So you kind of choose between high quality or fast. And even the fast model should supposedly be still pretty decent. Now, if you want to play with this, you can actually play with it for free right now. Over on Hugging Face, there is a hugging face space for Both stable diffusion 3.5 large and 3.5 large turbo. You can see here we've got an image of a capybara wearing a suit holding a sign that reads hello World. And as far as prompt adherence goes, it nailed all of those things. Let me try one of my long prompts that I've tested in the past and see how many of the elements it actually gets my old go to of a three headed dragon wearing cowboy boots, watching TV and eating nachos. Let's see how this does. Okay, so it took about 40 seconds. Here we've got the dragon. I'd say that's only a one headed dragon. They maybe gave it three legs instead of three heads, but we've got the dragon, the nachos, the boots and the tv, so it got most of it. It missed the three headed dragon part and gave it three legs instead. But not bad. Better than it used to do. Let's do a penguin in the snow holding a sign that says subscribe to Matt Wolf. See how it does with the text. All right, not bad. It got all the text in there. Let's see, did it spell everything right? Yeah, it looks like even spelled everything right. So pretty good. Now again you can Download Stable Diffusion 3.5 and run it on your computer. Just go to GitHub.com stabilityai sd 3.5 here and you've got your options to download with your install instructions and you should be good to go. It also works with Comfy UI it looks like. So if you know how to use ComfyUI.