How to scrape websites using Python and beautifulSoup

Hello and welcome to the channel again. So today we are working on scraping weather data from the National Weather Service, and we will be using Python to get this done. So this will be a very interesting project for you to work on. It's a very simple one. So before we start on that, first of all, I'd like to address a few things. Which is what you should consider before scraping any websites, and 2, how to explore the structure of the website that you're trying to scrape the data from. So just kind of walking through it. And then from there, we'll just import dependencies, get the HTML, find the outermost, and just kind of walk through all that to extracting the information and then putting it in a data— in a pandas DataFrame and doing some analysis with the data that you extract. So This will be a full rundown all the way from, you know, targeting which website to scrape, scraping the website, and then putting that data that you scrape into a format that you can do some analysis on. So stick with me here as we go through this. So for the first one, I want to start with what to consider before scraping any website. And there's so many things surrounding legalities and I guess regulations on how to scrape websites. But the main thing you should consider is, you know, any information that's public should be able to be scraped, should be scrapable, for example, in a way. Um, but if it's not, I would say please make sure you read the disclaimers in those websites on how they want the robots to navigate through their site, what portions of their site they're not— you're not allowed to scrape, basically. So What helped me a lot is a tool that I want to share with you guys. I was looking through my research and I found robots.txt. So if you look up, if you just Google robots.txt extension to your browser, be it Chrome, I'm using Brave here, which just kind of has a lot of similarities with Chrome. So once you add this up here, it lets you know, you know, if this website is scrapeable. And from what you can see here, all robots, you're not allowed to scrape. This part of this website, you know, and it will tell you. So if you see a website that just has this on allowed or disallowed, and it just has that, means you can't scrape anything from that website. But if it has— most of them give you specification, okay, what parts of the website you can scrape or which ones you cannot scrape. And you can read more into what robots— how to read a robots.txt file, and it will give you like how you should interpret it, how it's supposed to look like. And this Also goes into a little bit of detail, and you can— this developer did a good job of building this, which helps a lot in terms of if you want to scrape a certain website and you're not sure, you can use that tool to inform you on if you should scrape it or not, or what parts you can scrape. All right, to demonstrate this, let's go to the National Weather Service that we're trying to scrape, and let's see if they have any restrictions on scraping this site. And all I had to do is just run that, and from what you can see is robots.txt file not found. So they don't even have a robots.txt file on, on their site, which means they don't have any issues with you scraping it. But again, like I said, you need to go to the disclaimers for the site and try to find if they have any issues with you scraping their site. So, um, and that should give you some, uh, information about what what you can do and what you cannot do using, uh, web crawlers on their site. And you can read here, um, it has a lot of information. Go through it before you scrape any site. But I can tell you this site should be good for you to scrape since I've done the background research on, on my side. So you're safe to scrape this one. Second step I want to explore is how to explore the page structure or figure out what exactly you want to scrape from the site. So any website you can go to, you can do, you know, you can right-click on your mouse and click Inspect, and it will give you this kind of like development area here, console, where you can inspect each piece of the website HTML. So you can see this right here, it highlights the, the tag for the top and for the header, and it gives you all the information. So from there you can see the, the link to, to that and another link to the next thing, which is the second one. And you can even go further into it and it will even give you the picture that covers the top right there, which is highlighted. So that's how you can navigate this part, the structure of it. So let's just pick a city that we are interested in scraping data from and explore how we will be able to let our crawler find the information. So I'll put a random city. Let me do Blairsville. It's a city I've lived in in Georgia. And it's a very interesting, very interesting place. It's very dear to me. So let's do that. Blairsville, Georgia. All right, cool. And if you come here, what we want to scrape is we want to find this information. We want to extract all this data. We're going to extract the data that shows what the weather is this afternoon, tonight, Saturday, Saturday night, Sunday, Sunday night, Monday, Monday night. For this, we want to scrape all this and put it in a table. And we can manipulate it however we want down the road and do whatever we want to do with it. So the best way to do it is, since you want this data, you can come here, right-click, and inspect. And when you do that, it opens up exactly where you are, what you want, like the information on HTML part of this site that you need to scrape. So let's go and find this topic right here. So extended weather forecast. Extended weather forecast. So we can come to the content area. If you hover around it, it will show you what covers which area. And if you come here, you can see this is the 7-day forecast, which is what we are looking for. So the 7-day forecast, this is the first div tag. This tag here, div ID 7-day forecast, is what we're looking for. So that's the panel we're looking for. And if you click on this one too, it shows you the extended forecast for Blairsville. So this will be here, but what we're really interested in is this data right here. So this is the part that we want to explore and figure out how we're going to let our scraper walk down and find this information for us. So if you come to this 7-day forecast, that's what we want. And then under the 7-day forecast, you see this hazard one is the one that shows us the hazard here. But if you look here, these are all the information we need. We need the version for this afternoon, then there's one for tonight, all the way. So all these listed are all those different days that we want to scrape the information for, and they are under the class ForecastTombstone. So what we're interested in doing here is being able to create a scraper with Python and adding BeautifulSoup, guiding it to come find these classes and the data that is contained in there. So for example, let's look at this one, the first one, uh, which is tonight. So if you look at the tonight one, it's under the forecast tombstone, and then it's under the tombstone container class. And then if you click on it again, it shows you the period, and the period is this afternoon. So we want to have this data as well so that we know what's the time, you know, what, what's the forecast for. And if you look at— if you go further and look at the paragraph that's tagged here It explains, you know, this afternoon a 50% chance of showers, mainly before 3 PM. So it gives you all this information that we want to have, right? So that's how you— we will let BeautifulSoup help us walk down and find this information and scrape it and provide it to us. And this is just for this afternoon, so chance showers, and this is just a description, chance of showers. And then we also want to scrape the temperature, which is this right here. And that's repeated on all these other ones all the way through to Monday, I believe, or yeah, until Monday night. So that's what we were interested in. So let's go back to the code and we'll guide it through all this to find this information for us. Cool. Now the first thing we want to do is import dependencies. bs4 import BeautifulSoup. And I'll say this again so that you guys are tracking. So if you don't know what BeautifulSoup, you can look it up, or if you don't know how to download it, just look it up. You'll find tons of examples showing how to download this on Mac or Windows. It should be pretty straightforward. But if you have any issues, let me know in the comments and I will gladly post another help there to help you out. But those are the 2 modules that we need to import, which is BeautifulSoup and requests. So what requests will do for us, requests will go and grab all this HTML that's on this page. Like the HTML that made up this page, it's going to grab all of it. And then once it's grabbed all the HTML, we will use BeautifulSoup to sort out the data that we want from all that HTML that's been downloaded by requests so that we can store them in an orderly fashion that we would like to run with. Or do some analysis on. So I usually like putting this after the imports. Uh, it's not mandatory, but I don't know, I like doing it so that it tells me it's successful. All the imports were successful. So they're successful, I've already imported all of them. So the next step we want to do is get the HTML. We want to get the HTML using the requests, uh, like we mentioned earlier. So what we'll do is we'll comment out here, so we'll do What we want to do is ask the user to input a URL that they want to get the HTML from. So enter— ask user to enter URL to request. So URL. So how you ask user for input is you use the input function, and then in there we just write, you know, copy and paste a URL you want to request HTML. There you go, something like that, right? And what this will do is, once we run it, it will create a box underneath and it will ask us to paste a URL in there. So, and what I want to do is request the page, which is this one. We'll just copy this and and paste it once we get to that. But what do we want to do after we've put in— we want to be able to get that page and download all the HTML. So download, which is the next step. Download page containing weather forecast for Blairsville, Georgia. So what we want to do is we want to get the page And here we're using requests. So we already downloaded requests as requests. So you get requests.get. So we're telling requests, hey, go give me this URL, right? Which we already will input it. After that, we want to be able to view the HTML that it's brought to us. So we wanna prettify, and we'll have to prettify it so that it looks nice. So prettify_html in BeautifulSoup. So that's what we wanna use. Um, We'll use BeautifulSoup to purify the HTML so we can view it in a nice way. So we'll do soup, which is BeautifulSoup, and then we're telling it, hey BeautifulSoup, why don't you do this for us? So BeautifulSoup, get the page contents— content, sorry— and we want you to parse the HTML and make it look nice for us to, to view. So So HTML parser, that's what we wanted to do. So get the HTML that we were ready, and then what we want to do is we want to print the HTML, the soup.prettyfy, and it will give us all the pretty HTML that we need. So we can run this, and what we need is put the HTML. So we come up here, we've already inputted the city that we want, we just put here Blairsville, and this is the information that we got. And then, so it's correct. It has the longitude and latitudes for the area that we are trying to scrape from. So we'll copy that and come paste it in here, and then click enter, and boom! Voilà! You have all this information. So what it did, it went to that, it went to this site, and got there, used the using request. It got all the HTML, and here we're just using BeautifulSoup to Prettify the HTML so that it looks nice for our eyes. It's not jumbled up or anything. So this is how it looks. So this is all the HTML that we need, and all the information that we need to extract or scrape are all in here. So we just have to figure out how to navigate to where and, you know, extract it in a nice way. So that covers that. So the next step is we need to find the outermost element that contains the extended forecast data, and from there. The 7-day forecast. So this is what we're looking for, right? So we need to tell BeautifulSoup, hey, go find this tag on the ID div tag. Go find this tag for us, which should contain everything that we're looking for, um, in scraping. So let's do that. So let's guide it to that point. So we'll come here. So this is go find, um, We'll find the div tag. We'll find the div tag with 7-day forecast. Cool. So what we do here is, okay, let's instantiate a variable that's going to hold those instructions for us. So 7-day forecast, and we are telling, hey, BeautifulSoup, we which is soup. And then we're telling, hey, go find this. Go find this ID, which is equal to 7-day forecast. So that's all we're telling BeautifulSoup. Hey, BeautifulSoup, go find this ID tag that's labeled 7-day forecast. And when we come here, this is the ID tag that we're looking for here. So 7-day forecast. Hey, go find this in all this HTML. So we already got the request, right? Which is all this HTML. So we are already telling this BeautifulSoup, hey, go find that tag which is equal to this. Simple enough? Cool. Now let's move to the next step. So once it's got that, what do we want to do next? So we need to go find individual forecasts for the 7 days, or for, yeah, for the 7 days. Go find the individual forecasts. For the 7 days. Geez, my typing is terrible. All right, so once it comes here, we want it to come find all this. So all this— 1, 2, 3— all the way to Monday night. Go find all this. And how will it find out what this is? So what we'll tell it is, we'll tell it, hey, come here, go find this tombstone container which contains And all these have the tombstone container like you can see. So tombstone container, that class tombstone container. So under this tombstone container, what do we have? It has the period name, which is, you know, this afternoon, which is right there. You see, you can see it highlighting on the page. And it also has the, like, the extra information, you know, there's half a percent chance of showers. Sure, we want that information as well. And then it has a short description, which is chance of showers, which is basically a summary of what this is. in a short statement. And we also want the temperature, the high, the low. Yep. So we need to tell it, go find this Tombstone class and grab everything from all this as well, from all the days that we want. So here again, let's do forecast list. Go find us the forecast list. And the forecast list is equal to 7-day forecast, which we have already told it to go find up here. So it already has this here. So all we're telling is go up here again to this, the one that you already found, and we want you to find all this time. So find all that— find all classes that have this title and as their title. So tomb— tombstone container. So Go find all this, a class that has all this tombstone container. And then right after that, that means this forecast list has all the tags for all the information that we need for the 7 days. So for example, let's just test it out and see if it did what we needed to do. So sure, I'm going to show forecast for tonight. Show forecast for tonight. So we'll do tonight, and we know from here tonight is the second one, I believe. So this is, yeah, the first one, this afternoon. So yeah, tonight. So for tonight will be the second one. Um, if you know Python functions, uh, not function, but how indexes work, um, so the first one, if you do 1, if you want the first one, you just don't write 1, you have to write 0 because it starts at 0. So since it's the second one after this afternoon, it's 1. And let's just print that out to see how it looks like, right? If it did actually go there and grab the right one. So we'll do tonight. And remember to give it prettify so we can actually see it in a nice format. And let's run that. So from here, you can see it gave us some output. So it did go to tombstone container. And it grabbed everything that's within it. So it grabbed the period time, which is this afternoon. Okay, so something is wrong. Let's do 2. I think it also grabbed this and this counting. So we'll do 0, 1, 2. Let's see if that works. All right, so that worked. So tonight, if you see here, tonight, and it gave us all this description as well. The short— this is a long description, which is mostly cloudy all the way, and then the short description is mostly cloudy. And then the temperature low of 22. So it did grab what we want. So let's just comment that out because we don't need it. We don't need that at this point anymore. Now that you— we know that you can go and grab all this information, we want you to isolate and grab, you know, tonight, grab all this description. And we also want you to grab the temperature. So we have to tell it to walk down this as well. So the next thing we wanted to do is— the first thing, let's grab the period name first. Okay. which is this one right here, the class period name. So as you can guess, we'll do— go grab the period name. And from here, we just tell it instantiate as period. We give it— go to tonight since we already know you, you can find that. And from there, from tonight, find a class that is equal to period name right here. So go find period name, and from there as well, just get the text. We don't want— you don't want all the tags, all the HTML tags. Just go find the text. And then from there, let's also grab something else. Let's grab the description of conditions, you know. I Description of conditions. All right, so from here as well, we'll do a short description. And from there again, we guide it tonight. While you're here tonight, go find the class that's holding that. And the class is short description, short desc. And again, go find— just get text from this one. We're telling it to go find short description, which is this one right here. Go find this, and just— it should just return us most cloudy. And this first one, it should just give us tonight. And from there, what else— what other information are we interested in? We're interested in this information right here, which is mostly really the title here. So the title is tonight mostly cloudy Low with a low of 22 northwest and all this information. Just go find that. That's all we needed to go find. Extract title. Extract the title. And from there we can tell it title is equals to tonight's. We tell it find img. In this case, because it's under the img tag here. And then from there, what we want to do is find the description, which is this one right here. So all we're telling it is, hey, all this stuff that's been extracted from, from image, you know, go find title. So from image, find title. And just return that. So that should be that for us. Then the last thing we want to do is be able to extract the temperature, which we're really interested— low of 22, because I mean, you want to know the temperature, right, when you extract them. So that's some useful information that we would, we would like to know. So we'll do temperature, and from temperature, let's just do temp is equal to tonight again, and we'll give it find And this time it's class, right? Because to temp. And also this time just get the data, we don't want the HTML tag. See what, what we get from all this output. So let's print, let's print the period. We want to print the period. We also want to print, so let's first off First of all, let's comment this out because we don't want this to print again the next time around. So we already printed that. We need to print the short description. Let's print the short description in here, which is the description. And we also wanted to print the temperature. All right, so print temperature. What else did we miss? Short description, we already printed. So period, short description, title, title. Okay, so let's just run this. And something is not defined. Cool. So let's figure out what's going on here. Why are we getting this error? Let's just change this. I think this is where the problem is. So img, go find img, and then from that, you know, yes. So let's print it out. And it did. So I just mislabeled that. So I should have had it instantiated as an image. So it went here to the image. It went to tonight, found the image. And then from title, this is what it did. So it went— period is tonight. Short description is mostly cloudy. The long description is mostly cloudy with a low of 22 and all that information. And temperature is low of 22. So that's right. So this is valuable data that we want, right? So we've done the extraction of one piece of data for one day. So we want to do that for the rest of the 7 days. So this is where we'll replicate all what we did here on this extract all information. And once we extract all the information, we just combine that data and put it in the pandas DataFrame. So let's continue to the next step, which is extracting all the information that we need. So we've done it for one of these days or one of these periods. So we're going to do it for the rest of them. And this is how we'll do it for all of them. So first of all, let's select all items. So select, select all items with the class period name. So period name, and which is within the class of td. Tombstone container. So we want to grab all the period names under the class Tombstone container. So this is what I'm talking about. So under the Tombstone container, we want to grab this period name. And this Tombstone container is all this. It's inside of all these other days, right? So, but we want to grab the period name. So we want to grab this afternoon. And for this other one, we want to grab Under Tum Sing Konan, I'm going to grab period name. We want to grab tonight all the way through to— I think Monday night is the last one, right? So let's check here. So the period name for the last one is Monday night. Yes. So we just want to traverse through this entire HTML and find the periods, which should be all this up here. So how do we do that? This is kind of— you can see that the rip, how it works here with kind of what we've done in the previous. So we're still using Beautiful Soup. So period tag, grab all the period tags. So we'll do 7-day forecast, which we already called it up there. So 7-day forecast, and all we're telling it— what we're telling is go select tombstone, select tombstone container. And get the period, get the period name from each tombstone container. That's basically what we're telling it to do here. So this should hold all of our periods they want to grab. So from here, we just want to test because we just want to keep testing as we go. Want to test and see if it's going to give us any of those. So first of all, what we're going to do is I want to use list comprehension. So if you're not familiar with list comprehension in Python, I'll kind of go over it here in a moment. So, so we want to create a list comprehension to call the getText method, which will— and this is on each object, which will return, you know, tonight, Monday, Monday night, and things like that. So all the way. So So what we want to do is periods. We'll put it in a variable. And for list comprehension, you have to use the brackets. And then basically what you're doing, for example, you want to get the day. We're going to get the text for the day that says the day. If it's Monday night, we want to get it and convert it to text, right? As text. So, and then we'll do for day in tags, period tags. Tags. So basically what we're telling here is, hey, for, for day in all this, for day in all this, for each day, you know, go and grab me the period name. So grab me Monday, tonight, and all that stuff. So that's what we're telling it here in short. And we can print out periods so we can see how it looks like. So periods, and then we just do enter and you can see it. So now until 11 PM, this afternoon, tonight, Saturday, Sunday night, Sunday all the way through Monday night. So that's what we wanted to bring it for us. The other way you can write this, basically what expanding this list comprehension, basically this is what it is when you're learning. So for day, this is another way how it looks like on, you know, if you're not used to list comprehensions, this is like a for loop. So for day in peer tags, you got to do something, right? So return day.getText. So this is exactly what's happening when you do the same thing. And sorry, we can do print here. So Yep. So this, this is exactly what's happening here with, with this exact same thing. So it's just an expansion of how list comprehension works. The next thing we want to do is apply to the rest of the other tags, which is, you know, short_description, description, temperature, and to grab everything the same way. So Short description. Short description. We'll do so. T, I'll just go over this pretty quick, and I'll just forward it, you know, for the for the sake of time. All right, now that we have this, all this done, now let's just go ahead and print them out. So we'll do— we already printed out periods, so the next one is the short desks and desks and temperatures and All you have to do is enter and print them out. And from what you can see here, so we already tested that one out. So you can see here, it prints out the temperature. So let's just do one by one. It's probably best. So short description, as you can see here, is flood watch, chance of showers, all the way down. Now let's print. This should be the long descriptions. And you can see, you know, this afternoon it goes all the way into much detail. We know that works. So now let's go to temperatures and see. So it's aggregated all the temperatures that we need, and you can see all this, it's in a list of all the temperatures that we want. Next thing is to combine this. So to do that, first we need to import pandas because that's the module we'll use. So import pandas to create the data frames. So we'll do import pandas as pd. And here you can put it as import as x. It doesn't really matter. It's whatever flavor, but pd is very common. And people can understand that when they have to read your code. So create DataFrame. So now that we have Pandas, what we have to do is create DataFrame. So we can do— you can call your DataFrame whatever. So I'll call mine weather_df. So you know, we call pandas.DataFrame, and then we pass in a list, and the list is going to hold the column name and the value. So key-value. So the key will be the column name. So the column names are going to be all this. So let me just— so that I can see. So basically what we're doing here is we're taking all these variables here that hold the the list information that we need and putting it in a DataFrame. And each DataFrame, each column needs to have a title right at the top. So this is our DataFrame, and all we have to do is now is print the DataFrame so we can see weather_df. So most of the time, if you have a very, very large DataFrame, you can do— you can call heads and it'll print the first 5 values of your DataFrame. Or you can do tails, um, tail, sorry, and it will print out the last 5 values in your DataFrame. So in this case, we don't have much, we'll just print the entire DataFrame. So, and we're getting a ValueError. So let's see what's going on here. So we're having a ValueError, which is all arrays must be of the same length. So what's saying here, one of our arrays or a couple don't match. So how do we figure out which one is the shortest or doesn't have all the values? So we'll use print, and all I'm going to do is I'm going to print the values, the length of all the of all the variables, and we'll compare and see which one is the one that has a shorter or not matching basically the rest of them. And then let's print it. So from this, we can see that periods, short description, descriptions are all 9s, and the length for the temperatures is 8. So let's figure out why is temperatures shorter. So you can see 1, 2, 3, 4, 5, 6, 7, 8. So this one has 8, it's verified, but we're missing 1 and the rest of them have 9. So let's check out periods and see why it has 9, for example, right? So there you go. So you see it has now until and all this all the way to 9. So it has this afternoon all the way. Okay, so this one has an extra one, which is this one right here. Because if you look up here, um, if you look up here, the temperature 55 starts at index 1 on temperatures. But if you look on, uh, if you look on, uh, the periods, this afternoon is the second one, and now until 1 PM is the first in the index. So if you see here, this one has a value. All this has values as this temperature values down here, right? But this first one here, the flood watch, doesn't have temperature value. So we need to fix this. We need to add a value here in the database so that in the variable so that our data matches. So to do that, basically what we have to do is We'll use this 42 degrees Fahrenheit, which is the current temperature right now, and put it down here to represent the time now and the temperature that's representative of this time. So this, all we have to do is we have to call temps. We have to insert. We use the insert function here. We will insert 42 degrees at index 0. Should be entered to represent that, and it should fill in. So let's insert that. There you go. So we've inserted 42 as the index 1. So let's now verify that all our— and we can do that. So let's verify that all of them match. So they don't. Let's figure out what the problem is. Press Enter. Now they match, right? So it's 9999 all the way through. So let's go ahead and refresh our DataFrame here to make sure that we have data that matches. And there you go, so our data match. So what we can do here after we get to this point where we have our DataFrame, we can describe— for example, we can describe, find the description of our data in our DataFrame, how they look like. So we'll just get the description, description of data, description of our data. And this is very easy. All you have to do is do weather_df and colon describe. And this is just a pandas function that lets us look at our data in depth. So we can see here, you know, the count, 99 each on each side, which is correct, starts at 0. So So we also have unique values in each column. So the unique values here are 9. So none of these are the same. They're different tonight, Saturday, Sunday night, all day around. We didn't see what you can do on your part. So yeah, but this is basically how to do this using BeautifulSoup in short. So if you need me to walk you through the code or find my code, let me know. I can share my code as well. Hope you enjoyed this and see you in the next one. Bye.