How to Give AI Eyes: Unlocking Visual Data in Videos

Welcome to another snackable episode of the Business Lunch podcast. Normally it's me, Roland Frasier, and my business partner Ryan Deiss, but these snackable episodes let me share research I've been doing in a format you can actually listen to with the help of AI. So here's today's episode on giving AI eyes beyond video transcripts. Let's get into it.

You know, it is, um, it's really funny how easily we just accept a little bit of magic in our daily workflows.

Oh, absolutely. We do it all the time, right?

Like you paste a YouTube link into one of those, you know, AI video summarizers, you hit enter and a few seconds later, boom, you get a neat little bulleted list of everything that happened.

Yeah. And it feels completely seamless.

It does. And you probably sit there and think, wow, the AI actually watched that entire video for me. But well, I am here to completely shatter that illusion for you today.

It's a tough pill to swallow, but yeah.

Because it didn't watch a single frame. I mean, it didn't see the charts. It didn't, uh, observe the presenter's facial expressions. And it definitely didn't read the dense text on the screen. Not at all. All it did was quietly pull the transcript and just, you know, speed read the words.

Which is, to be fair, a really brilliant parlor trick.

Oh yeah.

For sure. And for a long time, I mean, we have all been perfectly happy to fall for it. But when you really think about it, relying purely on a transcript, it creates a massive blind spot.

A huge one.

Right. Because for a vast majority of the content out there, you know, software tutorials, complex product demos, heavily researched keynote presentations, Well, the spoken words are essentially just the background music.

Just the background music. I like that.

Yeah, because the actual data, the undeniable proof, the real substance, all of that is communicated visually.

Which is exactly why you are here with us today. Welcome in. If you are someone who, you know, wants to understand how things actually work under the hood and really how to get the most out of these AI tools without just falling for the marketing hype, this is exactly the place to be.

We're glad to have you.

Today we are exploring this really fascinating experiment outlined in an article from Crazy Experiments. And the authors of this piece, they set out to literally build, well, eyes for an AI, specifically Anthropic's Claude. Right, Claude. So it can actually process what is taking place on your screen rather than just what is being spoken into a microphone. Okay, let's unpack this because the mechanics of how they solve this inherent blindness are incredibly clever.

They really are. Yeah. And to understand why this experiment is even necessary in the first place, you have to look at a fundamental quirk in Claude's architecture.

Okay.

Claude cannot natively take video as an input. It just can't. You can hand it an image file, you can hand it, you know, massive text document. But if you try to hand it a standard MP4 video file, it just hits a brick wall.

It just doesn't know what to do with it.

Exactly. The architecture simply isn't built to ingest moving pictures. Now contrast that with Google's Gemini, which actually has this native video door built right into its foundation.

Right. Gemini can just take it.

Yeah. Gemini can take video frames and audio straight into its context window. Claude just doesn't have that native doorway.

And I imagine that is incredibly frustrating if you are someone who, you know, practically lives inside Claude's ecosystem all day long.

Oh, absolutely.

Specifically, if you are using Claude Code, their command line tool, asking it to just Hey, watch this video and tell me what is on screen. It's an impossible request. So if the doorway doesn't exist, you basically have to build one yourself. You do. And the authors of this experiment did exactly that. They stitched together 4 completely free open source pieces of software to create a custom Claude code skill, and they called it Claude Video.

And what's fascinating here is the sheer elegance of the workaround. Because the developers, they didn't attempt to like rewrite Claude's neural network to magically understand a video file format.

Right. That would be way too complex.

Way too complex. Instead, they approached it as a translation problem. They convert a format Claude inherently cannot understand, which is video, into a format it excels at understanding.

Which is text and images.

Exactly. A highly structured stack of timestamped images and text.

You know, it's It's like listening to a sports broadcast on the radio versus actually watching the game on television.

That's a great way to put it.

Right. On the radio, you hear the announcer say the score, you know, the quarterback threw the ball, but you completely miss the gravity-defying catch in the end zone, or, you know, the specific defensive formation that allowed it to happen in the first place.

You miss the context.

Exactly. You are getting the narrative spine, but you are entirely missing the physical reality. So how do we get that full television broadcast into Claude's text-based brain? Well, the first hurdle is actually getting the file. Right. They use this command-line utility called yt-dlp. It reaches out, downloads the video, and crucially, it grabs the native YouTube captions for free, assuming the creator actually uploaded them.

But that's the catch, right? Relying on native captions is a huge gamble because a lot of creators just do not upload them.

Yeah, they just rely on the auto-generated ones.

Exactly. So the tool needs a robust fallback, and this is where they introduce FFmpeg. Which is, well, it's a legendary open-source media processor.

It's everywhere.

It really is. The script uses FFmpeg to rip the audio track from the video, condensing it down into this tiny mono audio clip.

Okay. And if those native captions don't exist, we move to the next layer of the translation. The script takes that stripped-down mono audio clip and feeds it to an AI transcription tool called Whisper.

Right.

And I'm looking at the economics of this, and it is almost unbelievably cheap.

It's pennies.

Not even. They route the audio through Groq's Whisper Large V3 model, which costs roughly, get this, a fraction of a penny per minute. That's around $0.019. Yeah. Or they can use OpenAI's Whisper One model at less than a cent per minute. But again, if the YouTube video already has captions, this entire transcription layer gets bypassed, meaning the text extraction costs absolutely $0.

Which is amazing. But once the text is secured, we still have to solve the visual problem.

Right. The eyes.

Exactly.

Hmm.

So the script calls on FFmpeg again, but this time it commands the software to mechanically extract individual frames from the video and save them as standard JPEG images.

So now you just have a folder full of JPEGs and a long text file of the transcript.

Yep.

How does Claude actually make sense of that? Like, how does it know which picture goes with which sentence?

And that is where the clever prompting comes in. Yeah. The Claude video skill physically lines them up in the prompt that's being sent to Claude.

Oh, I see.

Yeah, it takes a block of text from the transcript, it looks at the timestamp, and it inserts the corresponding JPEG image file directly underneath it in the context window.

That's so smart.

It repeats this process over and over, essentially creating this synchronized multimedia flipbook. Claude reads the text, looks at the picture from that exact moment. Reads the next text, looks at the next picture.

It's forcing the AI to process the visual evidence right alongside the spoken narrative. Exactly. Now, theory is great, and you know, a multimedia flipbook sounds wonderful on paper. Mm-hmm. But I always wanna see the receipts. Naturally. Does this elaborate process actually capture anything that a basic text-only transcript misses? So to prove the value of this tool, the authors ran a head-to-head test.

Yeah, a real-world test.

They used a 14-minute and 45-second screen recording tutorial by by a developer named Nate Hurk, who is demonstrating how to give a powerful tool to Claude Code.

And we should point out, a screen recording tutorial is arguably the absolute worst case scenario for a transcript-only AI.

Oh, totally.

Think about the nature of a tutorial. The entire point of the video is the code being typed and the interface reacting on the screen. The narrator is usually just gesturing at the screen verbally, saying things like, as you can see here, or let's click on this.

Right, which means nothing as text.

Exactly. The spoken words are entirely dependent on the visual context.

And we see this play out perfectly in the first round of their test. They isolated just the first 70 seconds of the tutorial. Mm-hmm. The host of the video asks Claude Code to pull some quote-unquote wins from his school community platform. Okay. When the authors fed only the transcript to the AI, it caught the names of the community members: Michael, Chris, and Fernando. It also caught a spoken token, cost 260 tokens sent, 132,000 returned.

And that was it.

That was it. It missed absolutely everything else. Wow. Now I'm gonna play devil's advocate for a second here.

Go for it.

If I am a busy person prepping for a meeting, isn't a transcript usually good enough if I just want the gist of a video? Like, why do I need to know the exact file size of a JSON response or the specific dollar figures on a dashboard?

Well, if you're watching an entertainment vlog, sure, the gist is fine. But in a tutorial, a financial breakdown, or a dashboard review, the narration is not the primary data.

Right.

The voiceover is acting as a tour guide pointing at the real content. If you are a developer trying to replicate that environment or a business analyst trying to understand the financial stakes being presented, the gist is entirely useless.

It doesn't give you what you actually need to do the work.

Exactly. Relying on the transcript means you are actively choosing to ignore the primary data source.

And I'm looking at the results from the frames plus transcript method for those exact same 70 seconds, and the difference is staggering.

It really is night and day.

The transcript alone simply said, here is Michael's win. But because Claude could see the synchronized frames, it pulled exactly what that win was. A $31,000 AI deal plus a retainer.

It's incredible.

The second win from Chrisatsu was a €16,000 deal. The third from Fernando Gomez was €3.2K plus €299K monthly recurring revenue for an agency in Malaga.

And let me guess, not a single digit of that financial data was spoken out loud by the narrator.

Not a single digit. It completely flips the value of the summary from a vague overview to a precise data extraction.

And it gets even more granular with the technical details. You mentioned the token cost earlier. The host spoke those high-level numbers, but the screen actually showed a massive complex breakdown.

Right.

The frames allowed Claude to see that the response was a 529-kilobyte JSON file. That it wrote on a 450-character cookie. And most importantly, it showed that only about 2,000 tokens actually hit the context window.

Wait, really? It saw all of that?

It saw all of it.

The tool even caught the user's exact operating environment purely from analyzing the visual interface. Like, it documented that they were using VS Code running Claude Code version 2.1.133 on the Opus 4.7 model featuring a 1 million context window running Claude Max.

It saw everything on the screen.

It saw the entire project file tree structured down the left side of the screen. And again, the narrator never uttered a single word of that.

This test perfectly illustrates the massive data gap we just blindly accept when we rely on standard summarizers. We are getting the sanitized headline, but we are completely missing the underlying receipts.

Okay, here's where it gets really interesting. What happens when the transcript actively misrepresents the core argument being presented?

Oh, this is a huge problem.

In round 2 of their test, they jumped ahead to the 4:16 mark of the video. The presenter puts up a slide comparing two different ways AI can interact with your computer's files. Specifically, it compares MCP servers, which is the Model Context Protocol, a newer standard for AI tools against a traditional CLI or command line interface.

Okay.

Tracking. Out loud, the presenter says MCP used 35 times more tokens than the CLI and reliability drops from 100% to 72%.

I mean, if you were reading the transcript, that sounds like a solid mathematically complete data point.

It really does, but it's like reading the abstract of a scientific paper and completely skipping the methodology section where the actual proof lives. Yeah. Because the slide on the screen showed the detailed receipts under a heading called Schema Bloat. The slide actually read GitHub MCP equals 43 tools equals 30,000+ tokens before the agent does anything. Wow. The presenter never said that crucial data point out loud. He skipped right over the concrete numbers and just gave a sanitized ratio.

If we connect this to the bigger picture, this really touches on a fundamental human behavior and how we communicate in professional settings.

How so?

Well, when humans give presentations, we are terrified of reading our slides word for word.

Oh yeah. Death by PowerPoint.

Exactly. It is the cardinal sin of public speaking. So we put the dense, concrete, citable data on the screen for the audience to absorb visually, and we speak a highly sanitized, high-level summary.

That makes total sense.

We say schema bloat and 35 times more, but we show 43 tools and 30,000 tokens. A transcript-only AI is fundamentally designed to capture that sanitized summary and permanently discard the concrete proof.

So we are basically weaponizing our own presentation advice against the AI.

Exactly.

We've trained humans to speak in summaries to keep the audience engaged, and now the AI is getting trapped by that very habit.

It's a fascinating paradox.

At this point, anyone listening to this is probably sold, like you wanna give Claude eyes, but I am trying to figure out how this doesn't completely bankrupt you. If this is so incredibly powerful, why aren't we running every single video through this pipeline?

Well, we have to examine the hidden costs. And ironically, the true cost here is not financial.

It's not?

No. As we mentioned, the audio transcription is practically free and native captions cost nothing. The real budget you are spending here is context window tokens.

Ah, how does the math on that work? Like, If I feed an hour-long video to Claude, isn't it going to ingest thousands of images and immediately crash my context window?

That is the exact constraint the developers had to solve for. When you extract 80 frames from a video at the default 512-pixel width, you are eating up anywhere from 50,000 to 80,000 image tokens in your Claude context window.

Just for 80 frames.

Just for 80 frames. Images are incredibly token-heavy compared to text, so the tool imposes hard limits. It is not magic vision that sees every single frame like a human eye. Okay. It mechanically samples the video at a maximum rate of 2 frames per second, and it enforces a hard cap of 100 frames total.

Wait, that means if you try to scan a full 45-minute keynote presentation with a 100-frame limit, the tool's gonna have to spread those frames out drastically to cover the runtime. Yeah. You'll end up sampling what, 1 frame every 27 seconds or so? You are almost guaranteed to miss the exact moment the speaker flashes that critical slide with the methodology on it.

Precisely. You cannot treat this as a blunt whole video scanning tool. The solution is treating it as a precision surgical instrument.

I like that.

You use the cheap text transcript to find the general area of interest. Say you notice the speaker starts talking about schema bloat at the 2-minute mark.

Right.

Then, you tell the tool to densely sample a very specific narrow window, maybe from 2 minutes and 15 seconds to 2 minutes and 45 seconds, where you know the crucial visual breakdown happens.

Okay, that makes a lot of sense. But there is also the issue of resolution trade-offs, right?

Yeah, definitely.

The tool defaults to capturing frames at 512 pixels wide, which helps keep that massive token cost somewhat manageable. But at 512 pixels, trying to read tiny, dense code on a complex IDE screen is, well, it's a coin flip.

It really is.

The text can become a total blur. Now you can command the tool to bump the resolution up to 1024 pixels so it can clearly read the fine print, but doing that literally quadruples the token cost for every single frame you capture.

And that adds up fast. But to combat that token bloat, the developers included a really brilliant little feature using the dedupe flag.

Dedupe like deduplicate.

Exactly. This leverages an FFmpeg filter called mDecimate. What it does is analyze the pixel differences between consecutive frames. Okay. So if you are watching a talking head interview or a screen recording where the user doesn't move their mouse or type anything for 10 seconds, The screen isn't fundamentally changing, right?

It's basically the same picture, right?

The tool recognizes that lack of movement and automatically drops those near-identical consecutive frames. It flat out refuses to waste your precious tokens capturing 10 identical images of a static screen, which can cut your token costs by 30 to 50%.

Oh wow. That is an incredibly elegant way to preserve the context window.

Yeah, it's very smart.

So now that we've covered the mechanics, the test results, and the token economy, I want to talk about the user experience. Because this Claude video skill runs locally inside your terminal using Claude code, the workflow feels completely different than using a standard web-based tool.

It does. The output doesn't just dead end in a browser window. The video data becomes live in your terminal session. Right. You have the full transcript and you have the visual context loaded directly into Claude's working memory. Completely ready for you to interact with.

So, what does this all mean for you, the user? Picture your terminal window right now. You type in the Claude video command alongside a YouTube URL. Suddenly, instead of having to open a browser and sit through the video, your terminal downloads it, strips the frames, reads the text, and preps the data.

All in the background.

Yeah, it's like having a master sous chef in your kitchen. The sous chef perfectly prepares all your ingredients, you know, chopping the veggies, prepping the meat, so you, the head chef, can just step up cook whatever query you need.

I love that analogy.

You can instantly pipe that live data and say, summarize the on-screen steps into an actionable checklist, or extract every single dollar figure shown on that dashboard and format them into a markdown table.

You are fundamentally changing the nature of video here. You are turning it from a destination, something you passively have to sit through and watch, into a highly structured, queryable input.

That's a massive shift.

It is. But it is crucial to recognize that the best results come from the synergy of both streams. They cover each other's weaknesses.

How do you mean?

Well, a screenshot of a blank terminal window is completely useless without the transcript explaining what the host intends to type next. Right. Conversely, the transcript is useless without the frame showing the actual error code that popped up as the result of that action.

It is the perfect marriage of intent and result. Exactly. Now, just to manage expectations, there are scenarios where this specific tool will fail. Because the underlying script relies on YTDLP to download the source file, it completely fails on gated content.

Right. That's a hard wall.

If a video requires a user login, if it is region locked, or if it is a private corporate video sitting on an internal server somewhere, the tool just cannot grab it.

Which leads to a really pragmatic framework for when you should actually deploy this skill. Because you don't need it for everything. If your entire daily workflow already lives inside Google's ecosystem, you should honestly just use Gemini. It has native video ingestion built right in, so it is far less fiddly to set up.

Makes sense. And if you are just trying to pull the key arguments from an interview where, you know, two people are sitting in chairs talking, do not waste 80,000 image tokens.

ReadDoc.

A cheap standard text transcript will give you everything you need. But if you do your heavy lifting in Claude and the video you are trying to understand actively shows you critical information like complex code structures, financial charts, or software interfaces. No. Well, this skill is absolutely essential.

It effectively patches a silent failure that nobody in the AI space really wants to talk about. The immense danger of an AI confidently reporting on a reality it only half perceived.

We have covered so much ground today. We shattered the illusion that AI natively watches videos out of the box. We examined the massive quantifiable data gap between spoken summaries and visual reality, proving how you can miss $31,000 deals and complex software environments if you rely on text alone.

We really did.

We dug into the intricacies of the token economy, the mechanics of FFPEG, the necessity of precision sampling, and ultimately how to turn passive video into a structured, queryable knowledge base right in your terminal.

It is undeniably a massive leap forward for personal productivity. But looking at this experiment leaves me with a rather profound thought for you to consider as we wrap up.

Okay. Let's hear it.

We are rapidly moving toward a world where AI will be able to perfectly perceive and index the entire visual world. Every terminal command you type, every microscopic piece of data on a slide you flash for just 2 seconds, every fleeting facial expression. If we know that our primary audience for this content might soon be an algorithm extracting raw data rather than a human absorbing a, a spoken narrative, how will that fundamentally change the way we design presentations, build software, or even communicate on camera.

That is a wild paradigm shift to consider. Are we going to start embedding like invisible metadata in our presentation slides specifically for the AI to read, knowing the human audience will never even see it?

It's very possible.

Fascinating stuff. Well, thank you so much for joining us on this deep dive. Keep questioning the tools you use, keep experimenting, and most importantly, keep learning. We'll catch you next time.

Hey, business owners, I've got a quick question for you. Do you feel like you're missing the data you need to make strong business decisions? If so, it's probably time to build a CEO dashboard. It's an easy way to get everyone in your company literally on the same page, focusing on the numbers that matter. So The Scalable Company put together a free spreadsheet template that will give you everything you need to deploy your own dashboard. And to make it even easier, Ryan Deiss recorded a short training on how to use it. If you want to get your hands on the template, go to businesslunchpodcast.com/dashboard. That's businesslunchpodcast.com/dashboard, and you can download it for free.