Scrape Google for LinkedIn Profiles in Seconds with n8n

So let's hit test workflow. We'll see this process start. It's gonna go through for each of these requests. It'll go through into this code node. And let me drag in this Google Sheets real quick and you can see that we're gonna start to get all of these items put in. So there goes the first 10. You scroll down a little bit, we should see 10 more pop through. Now we've got even more items flooding through. It's in a total of 80 right now, 90 now. A couple weeks ago I made a video where we built this agent step by step and we gave it access to this tool that would scrape Google and it would return LinkedIn profile URLs based on, you know, how we chatted with the agent and what kind of profile we were looking for. And the video did well, but you guys were asking for what if we wanted to scrape hundreds and hundreds of profiles rather than, in this case, we were just scraping 10 because only 10 results show up on a Google search. So in this video, that's what we're gonna be talking about. If you haven't watched this original video, I'll tag it right here. I would definitely go do that real quick and then come back to this video. It'll make a lot more sense. But essentially, we're just going to be replacing this Scrape Google tool with one that, you know, has pagination in it so that we can go through multiple pages of Google rather than just returning that first page. So just a quick recap, here's the logic of that original tool that we had in the first video where we're scraping Google. It's really just an HTTP request, and when we run this, we get a huge chunk of HTML. And then in this code node, all that it was doing was parsing out just the profiles out of that massive block of text. And now we have these two different, um, workflows where basically it's the exact same thing. As you can see, the HTTP request, and then we're extracting the URLs, but we just have a pagination parameter before we go into the HTTP request. And then we're going to loop over multiple times based on every time we want to, you know, first time is going to grab the first page and the second page and the third page and the fourth page and so on. And so before we get into the actual configurations of these new nodes, I think we should just real quickly talk about HTTP requests and how all this works. So specifically when we're doing a Google search HTTP request, this is what we're doing. We, we start the URL with https://googlesearch, and when you type this into Google and you hit enter, this is what you're gonna see, just the classic Google search page that we all know and love. From there, we start to add parameters. So all that we're adding here is the parameter question mark Q. The question mark pretty much just states like, okay, everything after this is gonna be parameters that we're looking for within Google search. So we have Q equals site:linkedin.com/, which as you can see, this is what we type into Google. And then all this is gonna do is return only sites that look like linkedin.com/. So as you can see, all the results that are gonna be coming through here will just be LinkedIn results. From there, we want to, you know, key in even more based on different filters. So in this case, we're looking for CEOs in real estate in Los Angeles. So those are the parameters that we put in. As you can see in the actual Google search, that's what we're doing right here. And now we're getting back only search results from Google that are in the site linkedin.com and that also have hits in CEO, real estate, and Los Angeles. As you can see, Annie Chen is CEO, real estate Los Angeles, James Suarez, same exact thing. And it was just gonna go on and on and on because it's just essentially we're just searching Google. And then from there we get into pagination. So at the end of that, we want to throw in a start parameter. Start equals 0 would just be the first page of Google results, or You could just not have any sort of start equals and it would be the first page. And then you go start equals 10, which means the search is going to start after the first 10 results. And now you can see if you type that in, it's going to be on page 2. Start equals 20 would be page 3 and so on. So that's kind of how we're going to configure that pagination. So let's just get into how this is going to work. This is the first workflow that I was working on where we're just doing the same thing within this HTTP request. We're searching google.com, we're doing the site:linkedin, ceos, real estate, Los Angeles. Pretty much exactly what I was just showing you guys. And then I was testing out this thing where you can ask for a certain amount of numbers back. So we were asking for 100 results and then we were doing start. And this JSON start is just coming from in this pagination. Let's say I run this real quick. It's gonna give us 5 start values: 0, 100, 200, 300, 400. And this one would just be looping over each of those items. So it would first feed in HTTP request, and then it would fill in this parameter of start equals at 0, and then it would do it again for start equals 100, start equals 200, start equals 300, start equals 400. And that's pretty much what's going on here. And then let me just run this real quick and we'll see what happens. But I came into this issue last night where— okay, it's gonna work right now, so this is awesome. It's going to be getting pretty much like 500 results. It's gonna be extracting those URLs. So let's see what's coming out of this HTTP request. Like I said, it's a massive chunk of HTML. It's so ugly. It ran 5 times. So each, you know, this one I said was the start equals 100. This next one was going to be start equals 200. So it's doing a paginated request for us, which is awesome. And then what we're getting back, as you can see, is— okay, it's actually not working right now because another one of the issues I told you that I was running into. But basically, when you request too much, you're going to get some errors and Google is basically going to know that it's not a human doing the searching. So that is where we're going to start to run into issues. And that's where we have a different workflow that sort of uses a custom search engine in order to bypass this. So like I said, when I was doing this last night, I got super excited because I was grabbing 500 profiles at once and I knew I could have even made it more. So like, if you go into here, we go into the first run, which would have been the original page. You type in linkedin.com, that's what we're searching for. And if we go down far enough, we should see Annie Chen, which is actually the first search result that popped up right here. So we can see linkedin.com/anniechen. And back in these slides, as if you remember, when we searched for CEOs of real estate in LA, Annie Chen was the first result, followed by James Scott Suarez. So if we keep going down, we should see James Scott right here, James Scott Suarez. So we know that this is actually working. It's scraping, you know, 500 profiles. There just seems to be an issue right now with this, this code node. Okay, there. As you can see, I don't know what happened, but I just tested the step again and you can see that we're getting a ton of results back. So this first run, we got all these items, then we got, you know, all these items. You can see Annie Chen, James Scott, we got a ton of different profiles. It just keeps going on and on. So this is awesome and it's going to work right now, but if we keep doing this, we're gonna run into an issue where we're gonna get an error that's basically gonna say like you're blocked. So that is why we're going to move into the second workflow real quick that'll show another way to do this. All right, now this is the second workflow. This is, as you can tell, it looks pretty much identical. The only difference in this logic is that within this HTTP request, we're not doing a Google search in the URL, we're doing a search through a custom engine. So as you can see, it's www.googleapis.com/customsearch/v1, and then we put in our parameters. So All this really is, you're gonna come to Google and search custom search engine. You'll click right in here, programmable search engine, and then in here you're just gonna set up a search engine. It's super easy, it's gonna be custom, and then you're just going to want to grab right here your search engine ID. So this is the information that we need. And then the other thing you need to do is go into your Google Cloud. And if you don't know how to set up a web app, I've made videos where I sort of walk through how to set up all this and the consent screen, all that. I know for a fact I did it in the How to Create an Email AI Agent. I also just did it in my NNN Masterclass. So go through those videos if you get confused here. But NNN has docs, so it's pretty simple. But you just want to grab your actual API key within here because when you're setting up your custom search engine, this is pretty much how it's going to work. You're going to do HTTPS. You're going to do this, www.googleapis.com. And then once you start right here with the question mark, you're going to have to enter a key. That's where you're gonna enter your API key. I'm gonna put an and then you're gonna do cx= and that's gonna be your search engine ID, which I just showed you within, you know, the Google custom search engine. And then finally, that's when you're gonna start to add your parameters like, you know, job description, what type of job, location. And then you can also start to add, you know, start=10, start=20 in order to get those paginated responses. So as you can see, that's what we did in here. I'll have to blur this stuff out because it has some keys, but that's all we're doing right here. And then same thing with the json.start. It's the exact same thing in this one where we're testing this step. We're going to see what we're outputting, which right now we just limited it at 50, or I guess it's stopping at 40, but starting at 0. So the first one's going to go start equals 0, then it's going to loop again through the HTTP request, and then it's going to go start equals 10, and then it's going to go start equals 20. So it should be getting us multiple pages of results. So let's just test this workflow here. We'll see that happen in action. Start equals 0, start equals 10, 20, 30. It's coming out and now we've got these 5 different chunks of HTML, but it's coming through a little differently here. Um, we're actually getting the information over here in a, in an array called items where item 0, you can see here's the first link, and then we're going to be getting item 1. As you can see, it's a different CEO. We're getting the second link And you could change this if you went down here and you went to response. You could change the way you're getting the response. Like right here, it's auto detect. You could change that to text, and I think that's how you get a big HTML nonsense, but I didn't want to mess with that. We left it as is. And then when you come through here, that's where we're getting our LinkedIn profiles. So as you can see right now, we just got 50, but that's still pretty good. It's much— it's 5 times more than that first video we made. But obviously the code's a little different here. It's pretty much the same logic though. Whenever you're extracting something or sorry, whenever you're doing an HTTP request, especially from something like Google, it's gonna come back with some nasty stuff and you're gonna be able to work with ChatGPT to, in order like to figure out what you need to extract. So that's all that's happening here. As you can see, there's 5 different items, 10 items each. So essentially we got 50 LinkedIn profiles here. We could come in here and make the pagination limit higher so we could have this go all the way down to 100. And that is how, you know, we'd be able to get more responses. But from here, obviously the only thing we need to add is like the actual, um, this is the workflow that's grabbing the, um, LinkedIn profiles for the agent that we made. So all we'd be doing here is adding in the logic right here. So we'd be having an OpenAI model in order to variably fill in the parameters here, like how many profiles we're looking for, the job description, the location, and, um, the industry. And then obviously at the end, we just need to be filling in the Google Sheets. So we'd just be having these 50 items push into a Google Sheet after, or in this case, if it's 97 items, we'd be having those push into a Google Sheet after. So pretty much the same thing here. The only thing we did was add the loop and the pagination. So that's really what I wanted to show you guys is how that's gonna work, how you can have this increase each time, and you know how that's gonna go through into an HTTP request. Now we're back in that first workflow where we're actually just doing a regular Google scrape in the HTTP request just based on Google search. And I thought that I would run this again and maybe we'd get that error where we're finally, you know, sending too many requests and we're going to be getting blocked. So let's test this workflow. As you can see, now we got that error. It's saying the service is receiving too many requests from you. So this is exactly what happened to me last night after I was getting really pumped that I figured out how to get a ton of responses. But we're getting blocked. We're basically getting blocked because unusual traffic. So a couple workarounds, obviously the first one that I decided to do was just, hey, let's just go to a custom search engine. Obviously you're gonna have to start paying as you do more requests, but if you're getting a lot of, you know, leads, then I think it'll be worth it. But another thing that you could potentially try would be to add some sort of delay. So in between each loop, it may want to wait like 10 minutes, or I don't know exactly how it's going to work. So If any of you guys are more savvy with HTTP requests and sort of, you know, being blocked and stuff like that, then please let me know in the comments. But that's sort of the error that we were running into, and that's why we came into this other workflow where we changed things up a little bit by using a custom search engine. As you can see, this one's a little different. It's coming through a little different, but we're able to still, you know, parse out the information as we want. And, you know, obviously the end goal is just to get a lot of LinkedIn profiles. So as you can see, we're getting 50 items output here, and we would automatically feed these into an Excel sheet or some sort of document. We are now back in the custom engine workflow, so the one that I think is going to work the best. We won't be getting blocked on here, and I just wanted to change up a few things to show you guys. So in here, I changed the parameters to CEO, real estate, Chicago rather than LA, so we can just take a look at a different example. And then I added the Google Sheets on here so that we can see it automatically feeding through and sort of see how long that process takes. And then finally, instead of having this end at 40 for our start, we wanted to increase the limit so I can show you guys that you can grab 100 profiles just by clicking Test Workflow. As you can see, now we have these 10 items that go all the way to 90, and so this is gonna loop through and send a new request for all of those 10 start values. So let's hit Test Workflow. We'll see this process start. It's gonna go through for each of these requests. It'll go through into this code node, and let me drag in this Google Sheets real quick, and you can see that we're gonna start to get all of these items put in. So there goes the first 10. We scroll down a little bit, we should see 10 more pop through. We should see 10 more pop through as they get put into the Google Sheets over here. So this is going to go and it should give us about 100 because, you know, 10 items came through here and there's going to be 10 profiles in each one for the code node to extract. So we're at 50, we're at 50 now on this Google Sheets. So this is how it's going to work. Obviously, let's just quickly go into one of these profiles. So it should be a CEO in real estate, as you can see, greater Chicago area. Perfect. And now we've got even more items flooding through. It's a total of 80 right now, 90 now. So yeah, this is going to be how you're going to be doing it if you want more than just those 10 profiles. Obviously the next step here would be, you know, we add in the OpenAI node right here and we add in the when called by another workflow so that we can have this agent, you know, go pull those profiles when we say, hey, can you find a you know, VP in technology in Dallas, then it's going to be able to do that because you would have the OpenAI node be able to figure out those parameters and put them into the HTTP request for us. So that's how you can make that into an agent workflow. But for now, even if you were to come in here and manually just type in a couple parameters, you know, maybe make the limit of this 500 and have 50 different items here for your start values, and then it's going to loop through for each start value. That's all you need to do, and you're going to get leads super, super quick. And then also you're going to know a lot more about just how to sort of loop through and paginate different requests in order to find different information. If you're looking for research rather than you're looking for potential leads, or if you're looking for, um, you know, I don't know, really anything it could be. But being able to set up the base of how do you access a request, how do you being able to loop through so you don't get blocked because you're not, um You're searching with basically a robot, not a human. But really just wanted to hop in here and make a quick video because I know a lot of you guys were asking how do we search for more than 10 when I made that last video. So I hope that this one's helpful, and I really appreciate you guys, all the support lately. So thanks so much, guys.