Step by Step: Scrape UNLIMITED Emails for FREE with n8n

All right, what we have here is a trigger workflow that's going to take all 97 of these search queries, which are looking for sub-regions in California and looking for dentists, and it's going to run them through this workflow, which is going in and scraping them in Google, getting their emails, and then putting it into a Google Sheet for us. So let's test out this workflow and we can see it take place. I will also pull up the Google Sheet that it is populating this information into, and we can see this happen in real time. And this is all going to happen for free, and it's going to get you hundreds and even thousands of emails depending on how many queries you put in right here. But it's going to be free and you're not going to have to use any, you know, external APIs or use credits. As you can see, that first workflow just finished up, that first query. So we've got 12 in here. We can also go to our executions, and if this will refresh in a second, we'll be able to see that it's, it's triggering multiple. As you can see, the trigger went off and then it's going to scrape emails. This one just succeeded, and now it's doing the other one. And the trigger, you can see that we wait for— I have it set for 6 seconds to wait in between each execution. But this is just going to keep going and it's going to keep scraping emails. So I'm going to let this run. I'm going to let this finish up and I will just sort of speed this up so you guys can see that it's actually happening in this workflow and it's not me pasting in any information. So let's let this finish up. Okay. As you can see, the first 10 have finished up. We've got 159 total items have come back. And in this spreadsheet you can see that we have 159 emails in here, and we're going to continue to let this run. But before I do, I just wanted to say this build was 100% inspired by Akram. I'll tag his LinkedIn right up here and a link to his LinkedIn down in the description. So I found his build on LinkedIn and I, and I wanted to play around with it. So thank you for, for the build, and let's get back to the video. All right, just checking back in. You can see 58 items out of the 97 have finished. And we are at 814 total items here, 814 total emails as you can see in this Excel sheet. And so this will get us clear over 1,000 if we let this finish up. So let's just let it keep going real quick. All right, so went ahead and stopped it because we almost got to 1,000. Easy would have cleared 1,000 emails. As you can see, we got 909. So let's, let's quickly look into the executions here and we can see just every time that it ran, which would have been about 60 times if we remember correctly. So looking at all these executions, we're able to see exactly what happened and what was going through. As you can see, each time it was an individual different query. So each one was looking for a different sort of sub-region and scraping through. So if you want to understand how to, you know, set up this workflow and get it going, then stay tuned because I'm about to walk through step by step how to do this. If you're looking to download this N8N workflow that I'm about to show in this video, you can get the template in the FreeSchool community. So the link for that will be down in the description. You'll join this, click on YouTube Resources, and there'll be a post for each video I make. You'll just click into the post for this video and you'll see the workflow template to download, as well as there's a few code snippets later in this video, and I'll place the code snippets in here too. So I'll see you guys in there. All right, now to actually build this workflow, we need to understand a few things. So first of all, we're scraping these business emails from Google Maps. So in this example right here, we're scraping for the query of Bay Area dentist, and that search is going within Google Maps. So right here you can see in the HTTP request, it's going to Google Maps— or sorry, google.com maps/search, and then we're using the JSON query, which is Bay Area dentist, as you can see from over here. So real quick, let me just make this— let's throw this onto a new sheet, Sheet 2, so I can just show you guys how this is going to work. Hit Test Workflow. It's going to scrape in here from Bay Area dentist. We're just doing some logic here to filter the data— or sorry, organize the data how we want it, optimize the data how we want it. And then it's going to run through here, and this is where it's actually going to scrape out sort of those emails from each of those pages on Google Maps, the business on Google Maps, and then once again getting the data how we want it, and then it gets put into the sheet. In this case, we got 10 emails. So if we go to Sheet 2, we can see those 10 emails right here. So let's just get rid of these for testing purposes, and now we'll get into actually how do we build this. So the first thing to understand is we're searching Google Maps, obviously. So when you go to Google Maps, this is sort of the URL that you get. And this is what it looks like. But then when you want to define down and search, for example, in the United States where you want to look for, you search something like California real estate and it's going to pull up a ton of different ones, sort of the, the highest, you know, SEO businesses that are here, and it's not going to give you as many. So in this case, our request would have been like California plus real plus estate, because in an HTTP request you can't have blank space, you need to have the pluses to separate where space would be. So that is like the search that would be taking place. But what we're going to do is define down within California different like sub-regions or like sub-neighborhoods. So that's why here we went with Bay Area, because now we're getting more hits in this area. And then we're just going to iterate over multiple different sub-regions. So that's why in the demo you saw there were 97 queries that we were feeding in, which these were all just different sort of, you know, subregions right here. So once you understand that, now we will hop into N8N. Here is a new workflow I created for the purpose of building out this workflow, and then we'll also build out that trigger workflow as well. But we're going to start here. So we're going to obviously— this one, this workflow is going to be called by another workflow. So that's how we're going to start this one off. And for testing purposes, once again, we want to use Bay Area as an example. So you're going to come in here you're gonna type in query in quotes colon and then Bay plus area plus real plus estate. And so you could do this obviously with different sub-regions or different industries. It could be— in Akram's video, he did the example of dentists in New York, so he did, you know, different regions in New York, but we're doing California real estate. So now we have this data to test as it goes through. The next thing to do here is that we want to actually make our HTTP request. So within here, we're gonna keep it as a GET because we're asking for information back from a specific URL, which once again, we will go back here and we can actually just grab this part of the URL: https://www.google.com/maps/search/. So that's where we're gonna go. I'll put this in here. And so just to show you guys, if we copy and pasted this into Google, we would get, you know, Google Maps. So that's how that's gonna work. And then we just want specify what we're searching for within this search. So we're changing this to an expression, that way we can drag in this query right here. And obviously this is going to be dynamic based on what we put in the, um, the final test when we want to use that trigger workflow to send in— boom, Bay Area, boom, different query, boom, different query, boom, different query. And that's how it was looping over this workflow. And so that's really all we need to do. We'll hit test step to make sure it's working. As you can see, it got our ugly chunk of HTML data back. So that is why we need to add a code node for this next one in order to parse out the information we want. So I'm going to paste in a snippet of code here. All right, now I put in this code here. Once again, this will be in my FreeSchool community. Just look for YouTube resources, look for this video, and it will be in there. So now we can hit test and we will see what we get back out of this huge massive chunk of HTML. Now we're getting a ton of these sort of business domains. And as you can see, a lot of these are duplicates. These are kind of like, they're not the information we want at all. So that is why our next node is going to be to sort of filter out this information. So here we're grabbing a filter node, and within filter, all we want to do is we're going to be filtering this value, which is all the URLs coming back, and then we want to make sure that it does not match a particular regex, so a regular expression we're going to throw in this, um, query, I guess you could call it, or this information. And this is just saying that everything within these brackets— or sorry, within the parentheses— is what is like in this list of things that we don't want the URL to match. And then everything is being separated here by a— I forget what they call them— but the vertical bar. And so if we test the step, we will see that it's filtering out a ton of items. It only kept 32, which are going to be actual business domains that we want to email, although you can see there's some duplicates, so we're going to remove those in a sec. But as of now, in the discarded pile, you can see all of these emails that we don't want. So Google, Google User Content, Gstatic, gghtml-schema, all this kind of stuff. And just make sure you have this put in right. If you put like spaces, it's not going to work. So we'll test this again. And you can see now we kept some of those items that we didn't want because like the schema didn't work because we had a space in these ggpths. So make sure you don't have spaces in between, um, the vertical bars. And then if this isn't working for you for some reason, what you could do, and it's just a little more tedious, is you could do does not contain. And that way you could type in each individual one, and then you could go again and say, okay, it also can't contain you know, this information. But this is probably just the easiest, quickest way to do it. If you really wanted to, you could throw in like the, the.coms, although if you did it like this, I think you'd have to add this, um, you'd have to add this, this backslash in order to just sing— in order to, um, signify to regular expression that, you know, that's like how it would work. But you can pretty much just get away with doing it like this, so Hopefully I didn't spend too much time there, but let's test this stuff again and we will move on to the next part of this, which is wanting to get rid of these duplicate values. So we'll add remove duplicates and you want to do remove items repeated within current input, not these other two actions. So this is just gonna take anything from URL and you could just leave it as all, but I'm just gonna drag in URL. So anything in this field, it's going to get rid of that's a duplicate. So if you remember, we had 32 items right here. And now we have 20 items coming back because it removes those duplicates. Okay, now we want to do another HTTP request because right now we have these 20 different sort of domains coming back and we need to search through them in order to go get some more emails. So we're gonna do an HTTP request and we're going to be doing it from this URL. So it's super easy, we don't have to type anything out. All we're doing is grabbing this URL, but there's 20 coming back and we want to loop. So we want to add a loop, and then we're gonna have to get rid of this thing. And we don't want this to be the done field, we want to hook this up here. So it would have been smarter to just add a loop first and then HTTP request. So do that unless you're following this exact step-by-step, then you probably did that what I just did. But anyways, the important thing here to do is, as we know, it's going to loop through all 20 of these items. And what we can do within this loop over items node is make sure that it doesn't stop if there's an error in one of those URLs. Because currently all those 20 right here are going to parse through, and when it's being searched through the HTTP request, if any of them error, the whole workflow will stop. And so in the case of you running like 97 queries like in the example, the whole workflow would fail, or it would stop when it hit that failure. So we want to make sure that this is continuing, and we want to continue using an error output. So as you can see, anything that errors will just go down this branch and nothing will happen. We'll basically just skip it. And then we'll set up the logic for the ones that are successful. What do they do? So as of now, we are grabbing these URLs. I think that's all we need to do. Let's save and we'll test this step to make sure that everything's happening properly. So it just went through here. Now it's going to be searching all, um, of those items. Okay, so now we got an error and it did stop. So that's not what we want. We want to be able to keep going because it only did 5. So the resource you're requesting could not be found. Maybe this is the one as well. We need to make sure that this continues. Um, so maybe we can try this, test that again, see it coming through. And it was number 5 that failed. Okay, there we go. So now it would have just passed over the one that was erroring, and now we're able to keep going. So that's exactly what we wanted. And this one should finish up, and then we'll continue with the logic of these 17 runs that did work. As you can see, 2 of them actually errored, so that's fine because now the workflow is going to keep going because we have a done branch. And now we can set up this done branch workflow— or logic, sorry. Okay, we're going to drag this out and we are going to do another loop. And another essential thing is we want to make sure this does the same thing, continue if it's an error, and it's going to use an error output. So now we have an error branch here. And so we're going to get rid of this replace me, and we're going to add a code node. And let me paste in some code real quick. All right, here's our code. This one's basically just parsing out the actual emails into the format that we want. You'll see that in a sec. But another thing we need to do, the same thing here, is we— if, if one of these errors, which they likely will every once in a while, out of like, you know, 1,000 URLs come back, at least probably 2 or 3 are going to error. So we want to make sure it doesn't stop the whole workflow. We just want to make sure that it continues on throughout the logic. So I don't think we can just test this step directly because it's a loop. We have to run the whole thing again. So we will just test the whole workflow and then we can see everything going on. Um, once this finishes up, it'll come down to this loop and it will loop through this code node. And then I'll show you guys what I was talking about with the whole parsing emails. And once again, both these code snippets will be in the school, so join the link in the description. But let's let this one finish up and then it will come down here and this loop will be really, really quick. This is always the longest step is you'll be waiting for these to finish up. As you can see, that was super quick. So sometimes when you're running this at the end, some runs might take a minute or two, some will be 20, 30 seconds, and it's going to be waiting on this step right here. So let's look at this code node, see how this information is coming through. So it's getting this huge chunk again. The ugly chunk that comes through pretty much whenever you do HTTP request. And so usually you need a code node after to parse the information. So the 16th run, nothing came back. Here we got all these emails. So as you can see, it took this information and it got just the emails we want. Number 2, nothing. Number 3, that one errored. Um, 4, blah blah blah. So let's just see. So a lot of times this will happen like this, but anyways, you will get some emails. So Now we need to set up the logic of once we get those emails from this done branch right here, which are all the emails, what are we actually going to do with these emails? Okay, so we're going to drag out the done and we're going to grab an aggregate node. And this is going to be taking the emails that come back from the loop after it's been put through the code and parsed through. And then we want to actually just aggregate them together, but we want to make sure that we turn on merge list. And I don't think we can test— we're gonna have to test the whole workflow. Yep, we're gonna have to test the whole workflow. Okay, so we'll test the whole workflow and see how this information is coming back. We want to make sure it's in a list so that we can successfully split them back out. And it's hard on— it's hard to visually understand when I'm just explaining it, so we'll actually be able to pin this data and it will be nice because then I can show you. But yeah, just keep in mind when you're kind of testing these loops, you— it's, it's tough to pin the data, at least unless I'm doing it wrong. But now we have all these emails that are coming through in the— from the loop. And then we have them in one nice, um, sort of field right here. So that's why we wanted to do it as merge lists, otherwise it would still be sort of individual objects and that's not how we want it. So let me pin this data, and now that it's pinned here, this is going to stay, so that's good for testing. And we can add our split out, so it's going to turn a list and it's going to put them into separate items. And obviously we're just going to drag the list of emails. Don't drag one individual one. And then we should just be able to test it and it should be how we want it. Perfect. So we have these emails now because we can put them into an individual row within our sheets. And you might have noticed now that we have a couple duplicates, or more than a couple, we've got some duplicates. So pretty much the final step here is another Remove Duplicates. We're going to choose the first one, Remove Items Repeated Within Current Input, and then we want to select the field which will be from the split out node. We're grabbing emails, so we'll test this. We have 23 actual emails coming out, out of 246. So in this case, tons of duplicates. So it's a good thing that we did that, otherwise we'd be sending emails to the same, same inbox over and over and over. And then we just add them to our Google Sheets node. So we remember you want to append a row in a sheet. That's how you actually get them in there. We have signed in with our account, grab the right document, grab the right sheet, and then it's going to map them. We have a column in there called emails. When you're setting up your sheet to put stuff into, if you don't put a sort of like a column header, it's not going to work. So make sure you just throw something in there, that way you have a column to match on. And all we're going to do is send emails to the column called email. So this is what's coming through. We have our first example of an email, and if we hit test step, as you can see, it's gonna send those items into our Google Sheet. You can click into here, and there we go, 23 items just came through. Sorry, here's the one we're working on. So this is the one we just did, and 23 items came through. And so that's pretty much how this one works. Now we need to set up the logic of building that trigger workflow, and I'll talk about why we wanna do it like that. In order to parse a specific query through. So, you know, this one's going to Bay Area real estate, and then that one will run, and then the trigger will send a different query to the next workflow, and so on. So let's get into that workflow. Okay, so this one is being triggered manually, and I'll talk about some cool use cases as once you see how this works, I'll talk about some ways you can make this even more automated. But the first thing that we want to do is we want to grab a loop. Because we are going to obviously for, if we put in 100 queries to search through in California for real estate, we put them in here and then we want it to just send off, bam, first one, bam, second one, bam, third one. So let's remove this and we will grab workflow. Oh yeah, I was right there. Workflow, execute workflow. So this is the same way that an AI agent can call another workflow. You can also have a workflow call a workflow It doesn't have to specifically be an agent calling a workflow as a tool. So we're searching from our database, we're going to grab the scrape demo that we just built, and that's pretty much all we need to do here. We may have to change the way that it, uh, we should probably change on error. We don't— if one of the workflows errors, we wanted to make sure it's again continuing. So make sure you set up that, otherwise it won't run through all 100 items, however many you put in there. And then finally, we just want to wait. And so I did 6. I think you could 5, you could do probably even like 3. We'll try 3. Um, but the whole point is that you're— you just don't want it to— you don't want the emails to override each other in the, um, you know, in the sheet. Otherwise you'd just be putting information on top of each other if it was just sending out looping super fast. So That's why we do the wait here. And now we're pretty much ready. We want to talk about how you actually send off different queries into the workflow. So in here, I came into ChatGPT and I said, give me a list of search queries for the keyword real estate for 100 subareas, micro neighborhoods of California. And then you had to, you know, just show it how it's gonna be structured. So this is the one that I did in the demo. So we'll do a different one. We will ask for, For the keyword, let's try something like, um, finance. Let's just do 20, and let's do Chicago. And then an example, we will just say South plus Loop plus Finance. All right, so we'll send that off, and this should give us Wants to do. And so my, my thought was we're triggering this manually, right? So once this finishes up, I'll show you exactly what I'm talking about. But so we've got our 20, we're going to come in here and put in this mock data. So should be able to grab this, paste that in there. Perfect, no errors. And now we have these 20 search queries that it's going to send to Google Maps and scrape for emails. So let's just test this and see if I made any mistakes. And if I did, we'll live figure them out together. But another way you could do this that could be pretty cool is obviously I use ChatGPT to generate this stuff. You could map this up where, you know, you have logic over here where you can send the query or you can chat to some sort of agent right in here and just have it do it in here. So instead of going to— okay, so it only went through 2 items. Um, let's check our sheet. Okay, so we got a couple emails. Let's see what happened. So we'll go to our executions of the thingy, the scraper. So in here— okay, wait, what was the error? See, the first one came through. How many did this give us? This gave us 10 emails, which is, I think, is how many we have right here. Okay, we don't even have 10. Um, so what happened here? Maybe Lincoln Park Finance, that's the one that went through. Okay, let me just look into this real quick and I'll be back with what happened. Okay, I figured out what happened. The executions just didn't refresh, so we weren't looking at real information. But this is the first run. It gave us our 7 email addresses, which, as you can see, we had these 7. And then the second run, when it tried to go after it scraped through— what was this, Linkin Park Finance? It only had 2 items. And then from those 2 items, we only actually got 1 set, and then it just stopped because there was no more information. So maybe we need to make this where in here on error it would also continue, and on here it would also continue. So you want to make sure everything's still continuing, that way it will, um, it will always go through, if that makes sense. So let's also just unpin some of this data. Actually, that shouldn't make a difference, but we want to make sure every step is continuing, that way if for instance one scrape like Bay Area Real Estate, which in this case that one's fine, but the one that we just shot, I think it was Lincoln Park Finance, nothing actually passed through here, but we want the workflow to continue because that's the only reason this stops is not because it errored, but because it, um, just didn't have any data coming through. So we'll try this again, we'll see if that change worked, and then if not, we'll get in there and just look again what happened, and then If we still can't figure it out, we'll just search up or switch up the search term and go with something that works. Maybe, maybe finance isn't the best way, the best thing to look for on Google Maps. But there we go, we got our first 7 once again. Now we're in the second one and it stopped again. So we will look into that. So it's the exact same thing as what happened the first time. It gave us our 7, but then when we come to the second execution of Lincoln Park Finance. Once again, right here it ends. So maybe always output data. Okay, that's what we need to do. So much smarter than continuing. Okay, so always output data. So even if nothing comes through, we want it to still output so that it at least signifies to the trigger workflow that we've finished. That way It'll continue going for the rest of the queries because we've only gotten— we'd only gotten these 2 done. We still have 18 more to go. So, um, hopefully you guys like this sort of live thing where errors happen and we try to look into what happened and how to fix them. But, um, once again, this is going to scrape. Did the first one give us 7 emails? Hopefully the second one, it will give us nothing, but it will still continue to pass through the process. So now we're on number 2. Really hope this one continues to pass through. Please, please, please. We got our 7 emails from that first workflow, that first query. You would think that if no data is coming through, it would be quicker. Okay, perfect. So that seemed to be the issue. I'm glad we got that figured out live because I actually didn't, um, I hadn't run into a situation where that happened. So good takeaway here, making sure that if you're looping stuff through where you want the workflow to trigger multiple times, that you have, um, the Always Output Data turned on. And maybe that's like a super simple thing, but it passed my mind in this case. And now we're getting through the rest of our actual queries that we put in here, which we put in 20 queries. So let's let this one finish up and I'll be back. Okay, I lied, I'm back already. So we're on run 6, 43 items so far. I wanted to show you guys more about like if you clicked into down here all executions, then we could obviously see the trigger is still running because we triggered it. So it's been running for 2 minutes and then these have been all the iterations of the past queries that it's been feeding through. So this one's running for It's been running for 30 seconds now. As you can see, they're kind of gonna vary. So what was the first one of this run? We triggered this at 13:09, so 13:09 right here. The first one was 13 seconds, then 20 seconds, 20 seconds, 9 seconds, 30 seconds. I've had a few that will take multiple minutes. Some are as quick as, you know, like this one, 9 seconds. This one's going for, this one will probably push a minute. But it's cool because you can look in here and see how they're sort of staggered. And oh, I closed out of the trigger workflow, so we can't see that live anymore. But obviously this one is running, so we'll just come into here real quick, which is just one of the scraping examples, and we can look at the, the executions. So this one is— and we can see that one's going right now. This one was for Gold Coast Finance. For this one, we got 13 emails back, as you can see, and they were put into our Google Sheet, which is right here with our items. So yeah, I hope you guys found this one interesting. Um, that one just finished up. It took a minute and 26, and we only got like maybe 3 or 4 emails from that. But I hope you guys found this one interesting. Uh, once again, huge shout out to, um, Akram. Uh, his profile is in the description. But I thought this was a super cool use case of you know, having a workflow. And I thought this, this idea was super neat, the way that you iterate through items that you put here. And then once again, I think it could be really cool to, you know, you have some sort of OpenAI message model over here because that's essentially what we did in ChatGPT to get this stuff in the first place. You would have this, this over here and you would want it to, you know, have an interface where you could chat to it and then you could just ask it right in here to find you know, say I want to find places in, or I want to find dentists in Texas. And then this model would figure out, okay, well, based on how we prompt it, it would figure out to find sub-regions in Texas and then format them into this sort of query format with the sub-regions plus whatever the keyword was, which in this case I just said, I think I said dentist in Texas. So that would be really cool, and it could be an expansion on that. Um, so yeah, I might play around with that and see what I can do there, but that would even automate the process even more, which could be super cool. And yeah, any questions, um, please reach out, leave me a comment, or hop in the community, and I'd love to talk. So thanks for watching this one, guys. I'll see you in the next one.