How to scrape ANYTHING using Claude Code

What's up guys, in this video I'm going to be showing you how you can scrape literally anything using Claude Code by integrating the Apify MCP. And I'm also going to be showing you how you can build entire systems around these scrapers because we're not just connecting up the Apify MCP and asking it to scrape things one time because you could just do that through the Claude desktop app. With Claude Code we can actually build entire systems around these scrapers. That's exactly what I'm going to teach you how to do. So if you stick around until the end of the video, I'll teach you how you can connect up one single system and be able to scrape anything through Claude Code. Just quickly, if you don't know who I am, my name is Rory Rogers and I run an agency called Autonomous Lead AI where we find and eliminate operational bottlenecks in just 14 days to keep business owners working on the business rather than in the business. So if that sounds like something you might be interested in, then check the link in the description of this video and you can book up a call with me. Okay, so first of all, if you don't know what Apify is, it's basically a community platform of scrapers. So you can scrape literally anything on here. They're called actors, and like I said, they're made by the community And if we have a look through here, you can scrape anything. So Google Maps, TikTok website, Instagram, tweets, Facebook, YouTube. There is thousands and thousands of scrapers on here, and they're really easy to just connect up and use. But the issue is, if we wanted to use it through something like Claude Code, we would have to connect up each individual scraper that we wanted to use. So for example, if we wanted to do some work scraping Reddit, we'd have to go into here, grab our API key, figure out how to do the HTTPs, put that into Claude Code, just so it has access to this one actor. But by using the Apify MCP, it means that we can just put in our API key, connect up the MCP, and then Claude Code has access to literally any actor inside of Apify. All you have to do is ask it to use an actor. So that is where the Apify MCP server comes in that does exactly that. If you don't know what an MCP server is, it is essentially rather than connecting up individual endpoints from a system, in this case connecting up individual actors from Apify, Basically, they are all just packaged into one thing, and then you can just query the MCP server and have access to every single actor inside of it. So we just connect up the MCP to Claude Code, and I can have access to every single actor inside of Apify. So this is the GitHub repo here, and I'll leave a link to this if you want to check it out. And you can see some of the things it's compatible with. It is compatible with Claude Code, and it's super, super easy to set it up. I'm going to show you exactly how you can do it now. You don't even really need to have to come into this GitHub repo. I'm going to show you how to do everything. It's super simple, and the template for this workspace in Claude Code will be available, so you can just grab it and start using it straight away. Okay, so we're in Claude Code now, and I already have a project set up, and it's really, really simple. It just has an outputs folder, a scripts folder, and then mcp.json, which I'm going to show you in a sec, and then also a Claude MD file. So first of all, this mcp.json, this is how we set up the Apify MCP server. As you can see, This is the really, really simple JSON that goes in here that just says mcp-servers, Apify, and then the command in here, and then the arguments here, and then our Apify token. And all we need to do is just put in an Apify API key into here, and it will work. Because right now, because we have this mcp.json in here, if we hit forward slash mcp-servers, you will see that the Apify MCP server is connected. It won't work because there's no API key in there, but it's already connected just because of this mcp.json file. That's literally how easy it is. So if we come over to Apify, let's go to our settings here, API and Integrations, and then I'm going to copy over this key, put it into that JSON file. Okay, so that API key is in there now. Make sure to hit Command+S if you're on a Mac, and that just saves that file that you're in there. And then we can click out of that, and now we just need to reload this window here. So if I go Shift+Command+P, open this search bar up at the top, and then we just want to go Developer Reload Window. Reloads this to just give it a refresh for things like updating API keys or MCP servers or adding skills, etc. So now just to test that out, we can say to Claude check the Apify MCP server and give me the name of 5 Google Maps actors. So we're just going to send that off, and this is purely just to test that it's able to actually use the Apify MCP server. So it's just working away here. Apify search actors. From this, it looks like the MCP server is working, uh, because you can see it used an actual Apify tool here rather than just like searching the internet or something, and it's given us an output. We're not going to read through all this. It obviously is able to use the Apify MCP server. So that's literally it. That's all you've got to do to set it up. Add that mcp.json file and then just put in your API key and that's it. Now we're just going to say, okay, use one of them to find me 5 restaurants in New York. So it's going to pick one of those Google Maps scrapers and then actually use it through the MCP, and then it should just give us 5 restaurants in New York. So it's first using Apify Fetch Actor Details, so it needs to look at the parameters for the, um, actor that is used. So basically the things that it needs to input into this, probably things like location, maybe a star rating, the industry, which would be like restaurants. So it looks like it has everything. So it said, now I run the scraper for 5 restaurants in New York. Now it's using Apify Cool Actor. And then looks like it's just finished here. It says, here are 5 restaurants I found in New York. Uh, it's got the names, the category, the ratings, reviews, the price, the address, and then the phone number. Awesome. So it's able to actually use the Apify MCP server to do anything. And now we could say this about literally anything. We could say scrape Reddit, Instagram, TikTok, Google Maps. A website for us, absolutely anything. But what makes it different doing it through Claude Code rather than just doing it through like the Claude desktop app, for example, is we have the ability to do more than just say to it, scrape this for me, and then it uses one actor. We can actually chain them together and build external tools as well and just basically build an entire system around these actors. So we're going to start a new chat here and we're just going to say, I want to build a system using Apify actors that can find competitors in my niche and then scrape their websites and their recent social media activity on LinkedIn, Instagram, and TikTok. And give me a full report. So we're going to set this to plan mode here, and we're going to send this off, and it's going to come up with a full plan. It's going to get familiar with the Apify MCP. Maybe even in this plan it'll figure out exactly what actors to use. So it said, yeah, let me explore the current project state and research the available Apify actors for this task. So in the plan, it's actually going to come up with the exact actors, um, and then, yeah, it's going to give us a full plan here. We'll come back when it's done, and we'll either approve it, deny it, ask it to change stuff. Okay, so it's given me the full plan here, and basically what it said first is the actors that it's going to use. So the competitive discovery is going to be this similar web scraper here. And then for website scraping, it's going to use the website content crawler. And then for the LinkedIn post, it's using this one here. The Instagram post is using this one here. And then this is what it's using for TikTok posts. And then what it's doing is it's packaging these up into Python tools. So the first tool is going to be discover_socials.py, which means a Python tool. So it's going to use the LinkedIn, Instagram, and TikTok scraper, and then a Google search fallback, which is quite cool. And then it's going to use the generate_report function here where it's basically going to go through all of that data and give an executive summary, a competitor overview, traffic analysis, everything like that. And it's going to be done through a Python tool, which is quite cool. And then a competitor intel tool as well. So basically the orchestrator for all of this, which is really cool. And it's given us a directory structure here and then how the interactive workflow would go and also some error handling stuff and then the verification. So we're just going to hit accept this plan and then we're going to set it to bypass permissions here. Let this run through, and then I'll keep you updated if anything interesting happens. But likely we're just going to let this run all the way through, um, and then we'll come back when it's done. Okay, cool. So it's just finished now. It says everything works, all 3 scripts built and tested. I love the fact that Claude tests stuff for you as well. It's so cool. Um, so we've got the discover socials, extracts LinkedIn, Instagram, and TikTok handles from called website content using regex. Cool. Uh, generate_report.py generates a full report. And then the competitor Intel is the orchestrator with a CLI interface, and it just says to use it, give me your domain and I'll run the full pipeline: similar web discovery, website crawling, social handle extraction, LinkedIn, Instagram, TikTok scraping, and report generation. What domain should we analyze? So I'm not too sure here if it means give it my domain and it'll automatically come up with my competitors, or if it means give the domain of my competitors. So I'm just going to ask it here. I'm going to say, do you do you need my domain or my competitors' domain? Do you need my domain or my competitors' domain? Should be a quick question for it. Should come back pretty quick. Yeah, it says just your domain. The system will find your competitors automatically. Awesome. So let's give it my domain here. We're just going to use my actual domain here, and it'll come up with competitors for me, for my AI agency. Um, so put in autonomously-ai.com. And we're just going to send that off, and then we're going to run it through and see what kind of outputs it gives us. Okay, so it said let me kick off the pipeline starting with step 1, find your competitors via SimilarWeb. Awesome. So it's given us a full checklist here of step 1, step 2, step 3, step 4, and then step 5. Awesome. So it's, as you can see, it's really cool. It's using the Apify, um, actors in here. So it's SimilarWeb, found your website here. It said that it can't find any competitors on my website, so it's going to use the Google search fallback method. To basically just understand my niche and my industry and just search up some of my competitors. It's quite cool actually. In, um, Apify, we can check my runs. You can see all of the independent actors here running on their own, which of course been triggered from, um, Claude Code. It's really cool. It runs them all in unison. It doesn't do like one at a time. Um, bit of a limitation of course just with the plan I'm on. Um, it's using up all my RAM, which means it can only run 4 at a time. I believe it's trying to do 5. Yeah, as you can see, 4 out of 5 calls launched. Gumloop hit a memory limit. I'll pick it up when the other one's finished. So it's going to wait for all those 4 to finish, and then it's going to come back and do the 5th one, which is quite cool. It's quite cool. It does it all in unison rather than just doing them like one at a time. Obviously that would take way longer. Okay, so it's found my competitors, it's crawled all of the websites, and now it's moving on to discovering the social media handles. Should do that pretty quick. Um, yeah, I've identified the social media handles. That was pretty quick. And then now it's actually going to scrape their LinkedIn, Instagram, and TikTok content. This one might take a little bit longer just because it's, um, social media in general takes a little bit longer because there's more content. And also there's 3 social media platforms. So, um, yeah, we'll see how it does on that and we'll just come back when the HTML report is all ready. Okay, so I actually did it way quicker than I thought. It took way longer on the, um, websites and stuff than it did on the social media. But on my other screen here, I've just seen it's opened up an HTML report for me. So let's go through that together. Okay, so first of all, it starts off with an executive summary that just tells me how many, um, all of my monthly visits to my website my competitors. So it gave me 5 competitors, and then the social media presence found 4. So the competitor overview, it's given us a bunch of different companies here and done all of their monthly visits and the global rank, the bounce rates, and then page visits, average duration. Obviously can't really find that kind of data. It's also got me in there. Apparently I rank 3rd out of all my competitors, which, um, isn't bad. That's pretty good. I'm quite happy with that. That's the overview. And then the traffic analysis here Monthly visit comparison. As you can see, for some of these it couldn't really find their data, but for these, um, it could find all their data and made a nice little bar chart here. Website positioning analysis. I'm not going to read through all of this, but it's just given a positioning analysis of all of these companies here. As you can see, it's given me my one here as well. Awesome, awesome. There's quite a lot on here. And then LinkedIn activity. Apparently my competitors don't have much social media activity. You can see these guys only have one post and this is what the post is about. And then Gumloop obviously have a lot more here. Gumloop's obviously a software company, but it's in the automation space, so it's quite interesting that it put that on there. Um, these guys' top posts, the top posts, top posts, and all the themes, very, very interesting. And it looks like I couldn't find Instagram or TikTok activity for any of my competitors, which I guess, um, you know, is understandable. Sometimes they're not going to have TikTok or Instagram presence. If it's going to be any social media, it's likely going to be LinkedIn for any kind of like B2B automation agency or just B2B agency, um, in general. And then a competitive matrix. You can see Gumloop here, it's clearly taken the cake with, um, traffic and social media presence as well compared to everyone else. And the key takeaways and opportunities: no one has an Instagram presence or a TikTok presence, and Gumloop are leading in traffic by quite a lot. So you can see here how we've built a system using the Appify MCP that isn't literally just me saying to Claude, um, you know, scrape this for me, and it goes and finds an actor and scrapes it for me. Built an entire system around this with multiple different Python tools that all do different things, and they can even generate us a really, really nice HTML web page, um, to give us the results. That's the kind of thing you can't just do through Claude desktop app. You have to do it through something like Claude Code. So I hope you enjoyed this video, guys. As I mentioned, if you want to grab the free template for that workspace, you can just start using it straight away. Check the link in the description of this video and it'll be there for free.