Building an AI Agent to Edit Your Videos

If you've ever edited a video yourself, you know it's really time-consuming and can be very painful. It's so painful that it can kill your motivation to make videos at all. And I've tried every AI tool under the sun, including things like Descript, and the output is choppy and unreliable. My friend Shaw grew his YouTube channel from 10,000 to 100,000 subscribers in under 18 months, and he edited every video by himself with the help of AI. And it's really good, not slop. And the way he did it is using really smart constraints. So a specific file format, some transcription tricks that most people, including myself, got wrong, and also a way to steer the AI in a better language and with a better format. And so by the end of this video, you'll know all of them. All right. Yeah, I'm really excited to talk to you, Shah. You're a student in my evals class. That's not really important. What's really interesting is that you have an amazing YouTube channel. Has been growing like exponentially. Let's take a look at it. I don't know if you can share your screen, like we show like the follower graph.

Yes, I can.

Very impressive.

Yeah, I guess it's not quite exponential. It's, it's quite lumpy because it'll be exponential sometimes and then it'll linear other times. Yeah, let's see. Well, I've been doing it for a while, so it's like over 5 years now. So this is what the real data looks like. You know, kind of goes up and down. You know, sometimes it's, it's crazy. I guess the thing is you just have to post. That's like the downside of content. YouTube is better than some other platforms. Your videos, even if you posted it like a long time ago, it'll do well. Some of these videos I posted like years ago, but they're still like getting shared with people and redistributed. But like some of the shorter form platforms, they're not as good, but it's still a grind. You have to like You have to post every week or twice a week or whatever. So content is a grind. So I guess a lot of entrepreneurs or business owners or anyone trying to get into content, they kind of feel like it's magic bullet or something like that. And it can definitely be very rewarding. It's kind of like the only way you can get paid to get customers. At the same time, it is grind.

No, I feel you on that. Lenny Ratchetsky, he had introspective post and he said something like it's a treadmill and then if he's not on it, then everything, the music stops.

Yeah, maybe I can show you that because I stopped posting for 6 weeks and we can most definitely see a drop in my subscribers during that time. Yeah, so you see this period here, you can see I stopped posting videos. But yeah, so I'm getting thousands of views a day and then it drops down to cuts in half, almost a third when I stopped posting. Most of my videos are 20 to 30 minutes. And so end to end, making a video will take on average 10 hours, but it can sometimes take longer. I was able to kind of still get it done in a normal amount of time because I figured out how to get Claude to edit my videos for me. So instead of it taking probably 3 whole days, it took me a day and a half to get this entire video done, which is great, especially as a solo founder, just doing everything by myself.

So you have some kind of AI system in place that you're going to show us. Before the AI system, it was 3 3 days to create a video?

3 days. And then now it's definitely, it feels like it's cut in half and it's only gonna get better cuz I've already got ideas on how to automate other parts of the workflow. I have a good intuition of what's actually helpful to get me there. So just having ChatGPT write me scripts and then having ElevenLabs generate an audio of me and then having Gemini generate images or Sora generate videos and slapping it all together. Definitely there's probably a YouTube channel or a media company that you can build off of those tools. That's B2C focused or whatever it is. You can find that business that makes sense, but that's just not gonna make sense for me. And especially what I'm trying to do right now, I'm focused on B2B AI education. And a big part of that is people need to spend time with me. They need to hear me talk about things. They need to understand how I think about things to see if it's a good fit for the way they think about things and the way they see things and the problems that they're facing. You can't automate everything and it's not really about automation. So this is something I've just realized over the past few weeks as I'm taking Claude Code from being my software engineer for me to automating basically a non-coding task, all my non-coding tasks as well. It's not about writing prompts, it's not about building automations, it's about building system, building standard operating procedures, giving the agent the right context and giving it the right tools so it not only knows what you want it to do, but it has a better understanding of how you want it to get things done and making sure that the way it does work is actually aligned with what's helpful to you is actually in the same direction as your goals. You don't want Claude to just do a bunch of work and it going off in a tangent and maybe 10% of that work is helpful. You want 100% of that work to be helpful. And I guess the last thing I'll say here is if you think of it in the limit of how quickly Claude or ChatGPT can generate text. Now imagine that text was the source code and the SOPs and the playbooks, the marketing playbooks and the copywriting for a billion-dollar SaaS company. So in the limit, these tokens that the AIs are generating are very valuable, but you need to build the context, you need to build the harness, you need to give the agents what they need to build that stuff for you. Because out of the box, these AIs, they don't have goals, they don't have values. Humans have goals and values. So you have to provide that in the right way. And so that's why it's all about building systems. It's not about building automations.

I wanna dig into this distinction of what do you mean by building systems? Give me your prompt. I'll just call it copy and paste that. What you're saying is you have something very nuanced that you exactly go through.

And so all system is what's the process for getting something done? So I'm trying to get a video, I'm trying to make, post a YouTube video. What's everything that needs to happen so that gets done? I need a YouTube video idea. Where do YouTube video ideas come from? I need to talk to people. I need to talk to clients. I need to talk to prospects. I need to hear the language. I need to hear where they're at, how they're thinking about AI, what's confusing to them, what problems they're having. And this is the source of Inspiration. I just talked to a client, they're really excited about Claude agent teams. Oh, okay. So let me post a video comparing subagents to agent teams. So that's the seed of the idea. And then I run an experiment comparing the two things together, and then I form a narrative around that experiment. I guess this is the first thing worth calling out. So the first step in this system, this process for generating a video, it's all on me. It requires a human. To interact with reality, to experience things, and kind of be the seed for this creative process generating this YouTube video. Next step for me is to create a Keynote presentation. And so this is creating the slide deck. And so I could do this manually, but I actually have another agent skill for building Keynote decks, which has been super helpful. So there's actually a little mini process there. Okay, once I have the Keynote presentation, what do I need to do? I need to record myself doing a voiceover walking through the deck. And so that's gonna gimme 1 to 2 hours of content. Okay, so now that I have this raw video, what do I need to do to get it ready for YouTube? So I gotta do these cuts. So I do various cuts to get the content down, and then maybe I'll add some thrills on top of it. The final video will spit out of that process. And then I upload it to YouTube manually, and then I write the description. So there's actually a lot of details here, but traditionally this is me doing every single step of the process. Of course I'm using tools, I'm using Notion to kind of gather my thoughts. I'm using Keynote to make the deck. I'm using OBS to do the voiceover. I'm using Final Cut Pro to edit the video. And then I'm uploading it to YouTube. So it's humans plus software, humans plus traditional software to get things done. But now, how can we fit Claude into the picture? How can we fit AI agents into the picture? All this comes down to is just building agent skills. And so building these specialized agent skills that make sense in your unique and nuanced workflow.

I have a team, an agency that does my video editing.

Gotcha.

And I was having a discussion with them, hey, can you be more efficient with AI? And they were pretty adamant that no, it's not really there yet. We can't really use it for editing. And because you are doing it, you know it's possible. And plus they're going to be watching this video too.

This is a broader theme and you come from a machine learning background. So you've seen this before, which is to solve problems. In a business with AI, with machine learning, two key ingredients are required. You need someone who understands the technology, and then you need someone who understands the business problem. And rarely are those two people the same person. If you've ever met a machine learning engineer or data scientist, you'll know exactly what I'm talking about. Yeah. They're not the best communicators. They're not always super interested in the business. I think people definitely learn this over time, but early career, definitely not. Forget about it. But you need both. So I'm in this unique situation I think a pretty deep understanding of the technology, but also I have a very deep understanding of how I edit videos and my YouTube channel and things like that. So I have a deep understanding of the business problem and have a deep enough understanding of the technology where I can merge these things together to build things that someone who only has one or the other of these skill sets and kind of context, they won't be able to build it. So I think this is a universal principle. So it's, you need to bring both of those ingredients together to generate value with AI. So you could be a product manager, you'll need to learn a little bit to be dangerous. But if you're also coming from the technical background, you've put a cap on the amount of value you can generate if you're not willing to learn about a business, about an industry, or, you know, how to solve business problems. Surely there are people who are all technical, all gas. No business, and they're doing great, but they're the top 1% AI researchers, whatever, and they're getting the million-dollar salary or whatever. Someone who's really has very good product sense or really deeply understands an industry and they're able to make investments, big deals, and they can generate a lot of value with that. But most people aren't that. Most people are somewhere in the middle. So kind of figure out where you are and then fill in the gaps.

I agree with that. And because you have deep machine learning, engineering, data science expertise, from before and now you have edited hundreds of videos. And so you have spent the math amount of time to know—

Yeah, 1,000 hours.

Yeah, more than 1,000 hours.

Not 10,000. So I'm not the best video editor. I'm not the best data scientist, but I'm one of the few data scientists who've edited this many videos. And I think most people, if they reflect, they'll find that they're in a similar situation. They can stack their skills and be the best in the world. Or world-class at a very niche and specific subset of skills. If we can resolve that pain, or at least you get some inspiration.

Yeah, for sure.

Another thing worth calling out is that this is super valuable to me, but it'll almost surely not be as valuable to someone else unless their workflow is exactly mine, or at least very similar to mine. So that's kind of like an interesting thing with agent skills is that you can really optimize it to your world. And that actually makes it more valuable, not less. But I guess if you're trying to build a product, you're gonna have to take a slightly different approach. And that's why it's hard for Descript. That's why it's hard for Riverside and these other AI editor tools. They're building the product from thousands, tens of thousands of users. I'm just building this for myself, so it can be amazing for me. This video is how I automate work with Claude Code. I only did Macs for the experiment I was talking about earlier, comparing agent skills to subagents. I was like, I gotta go Macs to run these experiments. Experiments because it's very token heavy and I was going to cancel it. But then when I figured this out, I'm never canceling this. This is just a project level skill right now. Didn't even format the skill correctly, but it still works. It doesn't really matter because I explicitly tell it when to use it. But the thing is Claude wrote this skill by itself. I did not write any of the instructions. Okay, so basically this is what the skill file looks like. It's quite complicated. But the thing to point out—

I'm surprised that you said that you didn't write The skill. Yeah, you, I guess it's, you said like you gave it a lot of your knowledge, some baked it in. So where did that happen? Did you prompt it a lot to create this skill?

So it was an initial brain dump of what I'm trying to do. And so I just told it in English, my editing principles and basically what I'm trying to do and why I'm trying to do it and some principles. And then that was my starting point. The next step was. Basically having it apply that skill, those principles to a 6-minute clip. And then it tried to do it. And then I would, I opened up that clip in Final Cut Pro and just looked at it. And there was a lot of mistakes. I guess this is a little too far in the weeds, but you can cut it out if you want.

You kind of use a small video to iterate really fast. Okay, let me try it on a small length video.

Yeah.

And open it up and keep iterating until the AI is good at a small bit.

Yeah. So it was maybe 2 or 3 iterations back and forth on that 6-minute video. Just that was good. That was solid. And that saved me at least 2 hours of editing on my most recent YouTube video. It generalized to my latest video, which was, is the one we're doing right now, the one that we're editing right now. So then the next step is. I actually used it on this 1-hour raw recording, and then I went through and did the final cut myself. I refined its edits, and then I went back to Claude and I said, okay, these are the actual cuts that I did. Look at them, compare 'em to your cuts, and reflect on why I made the decisions that I made and distill them into principles and update the skill file accordingly. And so it comes up with a plan and then I manually re-read the plan, the principles that it got from it. And then I give it feedback and I usually, I do a lot of these things in plan mode. So I'm kind of in the loop of what it's thinking. Okay. And I give it feedback a few times.

The mechanics of the video understanding process from the LLM. Are you calling out to Gemini or something behind the scenes to look at the video? Is it just Claude looking at the video? Directly? Is the Python script, is it doing something interesting? How does it actually make the edits and know what's going on in the video?

So ultimately it's generating a transcript using AssemblyAI. And so I originally tried OpenAI by Whisper, but the problem is Whisper cleans up the transcription and you don't want that when you're editing. You want it to know where the ums, the uhs, the likes, the repetitions are. So it knows where to make the cuts. So actually the very first version using Whisper, the cuts were terrible and they didn't make any sense because Whisper had cleaned up the transcript a little bit. But AssemblyAI, their transcription model, it just does it exactly. So if you're building this for yourself, make sure you use an API, a transcription tool that keeps all the filler words and all the mistakes, because that's what you're trying to cut out. That's very important information.

There's a preprocessing step first, it goes to transcript, and then you're kind of operating on on the transcript itself more so than just the video.

Exactly, because that's way too many tokens. There are too many images. This kind of brings up that point where if someone is making very visual content, they're doing a house tour or something, this might not be super valuable to them. It's kind of my unique needs. The video is not super dynamic. It's just slides that I'm building in. It's mainly the voiceover and it's important The information is the important piece, not the eye candy, or it's not supposed to be entertaining content or something like that. And so that's why this skill may not be super helpful for someone with a completely different use case.

Where you're working in, knowing what constraints to put in is part of so much of the magic that makes things work. Knowledge that you have.

And I can show you this analyze.py thing. Okay, so it's done. So it took 5 minutes. Instead of the probably 2 or 3 hours it would've taken me to do this.

So it has some intermediate files. The JSON file is basically a manifest of its edits or something.

Yeah, basically. So we can see all the outputs here. Let's see. So there's a few things. So it did the transcription. So this is that test video I did. It was a 6-minute one. This is what we get from AssemblyAI, and so we have the full transcript there, and then we have the start and stop of every single word.

Okay, that's extremely detailed. So Claude is operating on your Python scripts and Claude are operating on this level of granularity. That makes sense, because if you're going to do cuts, yeah, you want extreme precision.

They're building a product for hopefully tens of thousands, hundreds of thousands of users, if not millions of users. It's just a lot harder to make something that's super valuable. It's just hard to build something valuable to a large group of people. Okay, so basically this is that transcript. And then what Claude does is it kind of codifies all its edits into this JSON file. Let's see, EDL video. So it's a text file used in video post-production that lists every edit. So I guess ultimately that's the source of truth or just a file that can be repurposed for multiple different formats. So that's exactly what we'll see here because there's a Python script called generate_fcpxml that takes this EDL and turns it into this specific format, which is FCPXML. So FCP is Final Cut Pro and XML is just XML file, which looks this. Again, this is machine-readable stuff, but basically this is a file you can upload into Final Cut Pro. And it has the edited video, but it's really important that the output, the artifact is this XML file as opposed to an MP4 file or MOV file. It's not helpful if it made destructive edits. If it cuts something out that it shouldn't have cut out, if it, a word got cut off or something like that, having the full printed MP4 file is not helpful to me. I need to be able to go through and refine the edits. And so that's why it's important to have it output as a, one of these XML files. So this is for Final Cut Pro, but I think it's also compatible with DaVinci and Descript and other video editors.

When you import it to Final Cut Pro, what happens?

So you can actually see this is the exact video that we're doing right now. I'll just import and I'll go to XML. So someone already went in and cut the video for me. So this is very helpful. Because I think we will probably see a good example of this. Okay.

Segment 1 and segment 2, there was something cut out in between the middle of those segments.

Yeah, exactly.

And how would you recover whatever is cut out between them? Is there a way?

Yeah, just this.

Oh, okay.

That's why the XML is so powerful here because it's just a text file that is referencing the source, the raw video file. And then the XML just gives Final Cut Pro the instructions on how to generate these segments from the raw video.

I see. Okay.

Yeah. And then basically it used to take me at least one cut to get to this point. So basically what Claude has done is cut out the filler, cut out the dead space, because sometimes I'll get stuck trying to explain something for 60 seconds, or there's some noise, there's a car driving by very loudly, and I have to stop talking for a second. So it's cutting out all that stuff, but more importantly, it's picking the takes. And so usually if I get stuck on something, I'll repeat it 7 times until I finally get the right wording. And Claude is cutting out those 6 initial takes for me automatically. And then sometimes maybe it cuts out too much, or maybe sometimes it keeps in a repeated take, but then it's really easy to recover. Or to remove something because of the format that we're in. So basically I jump to the final cut of the video or maybe the second to last cut. So I usually will do 3 cuts in a video. So this removes at least 1, if not 2 of the cuts. And it's just all quantified minutes.

I think the key was doing this on a short video first, refining it to be really good. And then you kind of had the confidence to say, okay, you know what? I can't, I trust it now.

Yeah, good enough.

Eval on a small video, essentially.

Exactly.

Okay.

Just getting it good enough. And maybe that's the equivalent of the vibe check. The vibe check, but instead of text input, it's a video input. The value of starting small is that it helps you go 0 to 1 quickly and you can iterate very quickly. I can watch that full 6-minute clip. So that 6-minute clip should have gotten cut down to 90 seconds or something. So it's really easy for me to evaluate. You'll get rid of those really big failure modes early on. And then I applied it to this 1-hour raw recording, which it cut down to 24 minutes, which is actually pretty close.

24 minutes, and I can see when you zoomed all the way out, there was thousands of lines. That's a lot of cuts. That would take you forever to make all these little cuts in this way.

It would just— it would have taken me at least 2 hours, and I feel I've done it a lot. If you're new to editing, it maybe would have taken 5 hours or something like that.

Wow. Okay. And so this is the first editing pass that happens?

The what?

Okay. What happens now after this?

Yeah. So now I go through and just fix it and then export it. That was basically the video. So I just had to do one manual cut, which is just making, just finalizing it and making sure it's good. But even that was getting to this point with the old workflow was hours of work, hours of editing. And lots of pain and suffering. But this is a lot easier. It's not just faster, it's also less of a mental burden. 'Cause yeah, I guess it's hard to describe, but those, the raw cut is a lot of thinking because you're making a lot of edit decisions. Do I keep this? Do I drop this? Or whatever it is. And you're making so many of them 'cause it's the raw video. But at this point, most of those obvious decisions have been made. So you're not wasting your attention on—

when you say raw cut, does it mean, hey, we discussed 3 subjects in this video, but you know what, let's just cut the entire third one out?

That, that it could be that, but it's usually, I guess if you could hear the audio, you would kind of get what I'm saying. But a lot of times with the raw videos, it's a lot of silence. How I'm talking right now, it's a lot of filler and things you don't want in the final video. It's not valuable. Audio. It's not valuable content. So cutting out all that. And also I'll have repeated takes very frequently in these raw recordings. And so cutting out the repetitions and just picking the one good take.

Okay. So the repeated takes, that makes sense. That's something that's dialed into your workflow. You do it on the fly and the way you make videos is you just repeat it until you get it. Some people have teleprompters and stuff. Yeah. Why do you think the— that Descript can't do this as good? This silence removal, filler word removal thing. That's their thing that they get so excited about promoting. But when I watch your videos, I thought you just did it. Maybe have a teleprompter. That's how smooth it is.

I think ultimately cutting out filler words and cutting out silences is relatively low value because most of the work is the repeated takes. Deciding what to keep, what to drop. And so if it, and that's going to be pretty unique to the creator or the type of content that you're creating. So I think it's just really hard for Descript to be able to build something that is helpful to everyone without having some kind of ingestion of their SOPs and their, for video editing or something like that. And most creators don't have SOPs for the video editing. It's all in their heads. It's all in my head. Claude didn't create the skill for itself, the principles wouldn't be written down anywhere. So it made this little editing principles.md file. And so this is basically all the principles I use for editing videos. This is just explained in Claude's words as opposed to my words. And so that's why I tend not to write prompts myself, cuz I think Claude just thinks differently than me. I want us to both look at reality together. And I want Claude to explain reality in its words, and that might be completely different than the way I explain reality. Oh, it makes sense.

I'm looking at— okay, timing rules: minimum segment length 0.25 seconds, avoid jarring microcuts. If I were to guess, I would say that did not come from your prompt.

Actually, a bit different. So I'll have Claude do its cuts, then I will go in and make my cuts and I just tell Claude, compare the two, look at the differences.

Okay, every single time?

Anytime I'm doing the iteration. Yeah.

So when you're doing this video that we were just showing, you went through the final pass in Final Cut Pro and then you correct Claude, maybe made some mistakes here or there. You would do this reflection and have it update the skill or what would you do?

Yeah, exactly. So you can actually export the XML. So let's imagine I just spent an hour just editing this.

You've added another constraint. It's not clear that a language model would be successful on the video. This form, you're looking at work in the XML, and now you've exported the XML. I didn't even know that there's XML.

Yeah, exactly. I didn't know either. I was talking to Claude about this and it said this one way to do it. It's this concept of agent empathy, putting yourself in Claude's shoes. And seeing the world from its perspective. And from that perspective, okay, how can I get it to do what I want in reality? Because the reality, you can look at it a million different ways. And so just the way I perceive reality and experience reality is very different the way Claude experiences reality, but we're experiencing the same reality when we're collaborating to get work done. And so it's just putting yourself into Claude's shoes and operating in that world and trying to think through, okay, what does Claude need to actually edit this video for me? So I could give it screenshots of images, but that's going to be super token heavy. Ideally, it's going to be this text modality. And luckily there's this FCPXML file, the standard that can be leveraged. And you can generate a transcript. And my videos are not very visual, so we don't even need the screenshots. When you really get into the weeds, it's pretty sophisticated, but I think there are a lot of high-level learnings that people can take away for building their own agent skills from this case study. I love it.

This is really cool.

AssemblyAI.

AssemblyAI, making sure you keep the messy stuff in there.

Exactly.

That's a brilliant insight. There's all these insights that stack up. It's pretty cool. And okay, maybe you can't, don't want to apply your exact thing, but I don't know, it's super helpful because maybe half of what you said, I was like, okay, you should totally know about this if you're going to do that, even if it's your thing is totally different. Just the XML alone, knowing that people can really grok that. Okay. That makes a lot of sense. Cause now you're operating in token space and that's something that LLMs understand. It can, this is way faster also, probably more efficient.

So I guess another kind of nuance that will come up too, it's not enough to write prompts. You also need to generate these CLI tools. Again, let's look at this. Look at this transcript. It has, we're not even at the bottom. It's 20,000 lines. It doesn't make sense to put this in the context window. You don't want an AI agent. You don't want an LLM to process this directly. You want it to build the right tools. The right command line tools so that it can process this information in a way that makes sense. So you don't blow up the context window. Yeah. This one operates on this EDL.

Yeah.

So it's the EDL is already pretty ref— actually, no, it's 1,600 lines. You don't want to put that in the context window either, but yeah. So let's take a look at it. This converts the EDL to the Final Cut Pro XML. I haven't looked at this code. This wasn't what it came out with on the first pass. It, this is after multiple iterations and after editing the 1-hour video, but basically it's doing a combination of things. It's using a combination of heuristics to trim down the video as much as possible. And then generating the transcript and then feeding that transcript into the context. Okay.

Yeah, it's using heuristics, I assume for silences and stuff like that. It's heuristics, which makes a lot of sense. You don't want to throw AI at every single nook and cranny. It doesn't even make sense.

You don't want to put everything in the context window.

Yeah.

Ultimately, anything you don't have to put in the context window, don't put in the context window. Just because those tokens are precious.

Were you telling Claude it needs to make its own tools very specifically? Or was it, hey, I need to make tools and it made its own tool?

I guess the kind of approach I took is kind of coming in with the mindset of, hey Claude, this is what I want to do. What would it take for you to be able to do this? For me, or would be able to get this done. And a lot of back and forth, even though I didn't write a single line of code, it still requires some technical background. So you kind of know what kind of makes sense and what doesn't make sense because it's using command line tools, FFmpeg. And so if you've never used FFmpeg before, then you don't know if that's— maybe it suggests another command line tool that's not as well established. And so you can't make those technical decisions if you don't have the background. So yeah, yeah, there are some landmines that can come up when you're trying to build this stuff, but it's just amazing how it probably took me 30 minutes to make the initial version of this. And I've just been iterating on it as I go. So I'm not spending dedicated time to improve this. It's basically, I'm just using it and every time I use it, it gets better. So because you have to do the diff, compare the diff. Exactly.

XML files and you say, is there any principles you can extract from that?

Yeah, exactly.

And so do you think it can go further? Okay. I know you say you do a raw cut first, then you put it into this.

I don't do the raw cut. I just record the video and have Claude basically, because before I would do a raw cut and then a mid cut and then a final cut. Okay. Now basically I'm trying to get Claude to do all the cuts and then I just do the final cut basically. I see.

Do you think it can do more editorial style things to say, hey, you know what, this introduction was too long. If you just cut out this part, it'll go smoother. That kind of decision going into the joining you in the more editor level.

If it has enough data, then yes. So a big mindset shift for me recently as Harpothes Software 1.0, 2.0, and 3.0 and things like that.

Yeah.

I used to kind of just say Software 3.0 is just LLMs. So I kind of lumped a lot of in, but more recently, and maybe this was obvious to other people, but maybe it took me a little longer to figure out, but to me now, Software 3.0 is all about in-context. Learning. So I used to kind of lump fine-tuning in there as well, but I don't know, that kind of, that unlock was very helpful for me. So how do you train it in context? You can include examples in context, but you can also distill principles from examples. So if you give it enough examples and it distills the right principles from those examples where you're basically giving it examples of editorial choices, and so since I have 150 YouTube videos of the raw video and then the final cut, the thing I actually posted on YouTube. That's probably got everything that Claude needs to know about how to edit my videos.

And you can do the, get the XML from both, I guess. And you can do that. Have you done the exercise? The 150? Go through that and see.

So that's basically what I'm on now. So if you look at the repo, I've got this evals folder. And basically what I've got here, I took 3 videos. And I basically generated a bunch of clips from these 3 full-length videos. These are all raw videos. And now I'm building this eval benchmark.

That is super cool, man.

Yeah, so now—

You can have 150 pairs.

You can have 1,000 pairs. I take the full video, so that's one thing, but then I have Claude use some heuristics to take that full video and turn it into smaller clips. So one video becomes 6 mid-sized clips, 5 to 10 minute clips. And then I have it do Shorts too. So this one video turned into 1 plus 12 plus 6. So 19. So I got 19 videos from one full video.

You mentioned Shorts and this is not probably what you were talking about, but do you do any kind of short form videos, splice your long form and create Shorts out of them ever?

I used to. When you're a solopreneur, it's all about leverage.

Right.

And I just saw that it just wasn't making sense. At one point I was posting everywhere. I was posting daily on Instagram, on TikTok, on LinkedIn, on YouTube Shorts. It was a lot of work for not a whole lot of impact. So I found the highest ROI thing is just to make long-form YouTube videos. And that's just been the main focus.

Makes sense. That also makes sense to focus. There's a lot of leverage comes from focus on something.

That's cool.

When you get these eval set, we should get you in the class.

Yeah, I'd be happy to. And then I guess you'll have an example of running evals on video data, which no one thought was possible until today.

That's super cool, man. Yeah, if it gets good enough, you should totally think about making some kind of product for this. This is a real pain point. Getting people to do this and having it train itself to be better. I can't think of any product category that's doing that. Maybe coding agents are doing it at the foundation model level. That's, I think that's the only category that's trying to do that.

So two things here. It's kind of a flavor of reinforcement learning. So that's what the cursors and ChatGPTs and Claude's of the world are doing. They're updating the model's weights based on how users are actually interacting with their models. That's reinforcement learning, but this is a little different. This is in-context reinforcement learning because we're not touching the weights. We're just changing this agent skill.

Yeah.

And so that's actually a lot cheaper and much more flexible. So you're not just printing this neural network. That's this fragile thing. I guess you could have a lower weights or something adapters or something like that, but still that's kind of overkill. It's kind of underutilizing the main power of LLMs, which is in-context learning. So I guess I'm just now being awakened to how far you can go. I was awakened when I took your course, but now it's a second awakening with these agents. Now you have the data.

So you have this dataset and now that is super exciting. That's something that you can iterate on. Very cool, man. Where can people find you?

Best place is YouTube. Just go Shaq Talebi on YouTube. I've got a ton of free videos, free content. So that's probably the best way.

Sounds great, man. Yeah, I learned a lot from this. This is so cool.