Web Scraping for LLM in 2024: Jina AI Reader API, Mendable Firecrawl, and Crawl4AI and More

Welcome to the second video of our series on data scraping. In the first video we looked at a tool for parsing complex PDF files. And in this video we will look at a number of different tools, including open source, free and paid, which will help you scrape data from web pages. Now, as we know, LLMs require a lot of data and most of this data comes from web pages. However, the problem with the web data is it's really messy and unstructured. It contains a lot of noise, there are inconsistencies in formatting, and there is a lot of irrelevant information. To parse this correctly, you usually want to convert HTMLs into markdowns. So traditionally people have been using tools like Beautiful Soup, which extract contents based on HTML tags so you can retrieve tables, images, links and so on and so forth. However, you normally need to create very complex handcrafted rules in the form of regular expressions to retrieve content efficiently from web pages. Now that brings us to the use of LLMs to process HTML documents. Now there is structured in HTML files, so you can actually train LLMs to understand these HTML tags and structure and then use it to efficiently retrieve data or content from HTML files. So the goal of this video is to show you a number of different tools. We are going to look at completely free but not open source. We're going to look at some paid tools which will let you scrape data from websites. And we're going to also mention a few completely open source tools that you can use to get started with data scraping from HTML documents. So let's look at the code. Before showing you the code, let me walk you through these example web pages that we're going to be using for web scraping. So the first one is going to be this blog post from Hugging Face called Training and Fine Tuning Embedding Models with sentence Transformers v3. This web page has a table of content, then there are headings. They also have some code segments and I believe there is a table as well. So this is basically a type of webpage you will come across if you are trying to scrape blogs or tutorials. The second one is going to be a bit more complex. So we are looking at an archive paper and it's the HTML version. So again we have a table. There are some images in here and, and there are also a number of mathematical equations in here. Now to make it more interesting, we're going to be looking at a PDF version of the same paper because a lot of these web scrapers that you use actually fails if you're trying to retrieve information from A PDF which is hosted on a website. Okay, so to get started we are going to first look at Beautiful Soup as our baseline. So this is the traditional approach. We will need to install the request package as well as the Beautiful Soup package. And then we're going to use this python function to scrape data from a given URL. So first we import the requests library, then import the Beautiful Soups library, and we'll need to provide a URL of a webpage. We'll use the request package to actually read the data and then use the Beautiful Soup package to parse it and give us a final string output. All right, so the first test is going to be using that hugging face blog. We pass on the URL to the function that we created. We'll get data in the form of a string. And here is how the output looks like. So as expected, it's actually an HTML code. There are tags for different elements on the web page. Now you will need a post processing step. So you will need to use regular expressions to actually extract data or content from this web and that can vary depending on how complex the web page is. You will actually need to come up with handcrafted rules that are going to be implemented using your regular expression. Now as a second test, we will look at the PDF file that was hosted on a webpage. So we provide the URL of that PDF file. Again, we're going to use the Beautiful Soup to scrape that data. And here is the output. So you can see it's completely messed up. It's not able to decode the unicodes in here and it's going to be extremely hard for any LLM to work with this data directly. Now to deal with this next I'm going to show you a solution that is probably one of the easiest to use and has the best output. And this is called Reader API. It's from a company called Jenna AI. They are not sponsoring this video, but I really like their tool and it's ease of use. Alright, so all you need to do is to append your URL on top of this base URL or R Jenna AI and it will be able to scrape that web page for you. Now it does a lot more than web scraping and I'll probably create a dedicated video on different features that are available. Now the best part, you can use this state of the art tool for a price of free, which is kind of crazy to think about that the company is shouldering all the cost. Now there are definitely rate limits. So if you use it without an API, the rate limits are lower compared to if you use it with an API key. But even I think the API key is free. So here's just an example and you can actually provide your own links here to see how it's going to work. So for example, here's a Wikipedia article. This is what the raw HTML looks like. So this is what you're gonna get as a result of using BeautifulSoup. And here is a pretty nicely formatted version if you try to use the Reader API from Jenna AI. So I copied the blog post and here's what we get out of raw HTML and here's the output that you will get if you try to use the Reader API, which is pretty nicely formatted. Now they also give you the ability to actually ask subsequent questions. So you can do rag on the contents. But let me show you how to repeat this whole process in Python code. And for this we will just need a single command. So again we're using the request library. This will read the pages, but rather than reading the base URL, we are adding this URL at the beginning. So exactly the same way as I showed you on the web page. And we're going to be using this function. So you provide the URL. If you print this, you get a pretty well structured HTML file and this is markdown. So you get the title, then the source. This is, this is what you're going to be providing and then you will get the markdown content. So here we have the table of content, then different sections and everything is pretty well formatted, which is pretty nice. Even the code segments are well formatted. So it's, it's pretty amazing because you can use this for free without writing a lot of code. Now another great feature that they offer is you can also read the contents of PDF files. So in the next test we provide the PDF URL for the paper and you again get a pretty well structured markdown as a result of this API call. Here are even the latex equations that is able to preserve. So overall pretty interesting. As I said, the Reader API from Genai is probably one of the best and the easiest option to work with. But there are also some other cool solutions. So for example, there is one called firecrawl. This is from a company called Mendable and they give you free credits. The great thing is that you can run this locally and let me know if you are interested. I'll create a dedicated video on this. Now, if you want to use the hosted version, there is a free plan that will give you 500 pages per month and there are some paid plans as well. But I just want to show you how to get started with this. So here they have their own playground. Now you can either use Crawl or you can use a single URL. There is also a feature for LLM extraction, so very similar to what we saw for Gen AI. You can actually ask, question and retrieve information. Now to get started, just provide your URL, click on run. This takes a few minutes but it will scrape that data for you and you will get a pretty nicely well formatted markdown. So very similar to what you will get out of Jenna AI. And if you click on get code, this will give you the code that you can use for scraping web pages. Again you will need an API key, so you actually need to set up an account in this case, but you get some free credits to get started and then if you want higher rate limits you can always sign up for their paid account. So here is a quick code examples of how to get started with Firecrawl. You will need to install the API client. Then you will need to set your API key that is going to come with the account when you sign up. And then you can use a function like this to basically first create the client, provide your API key, then you provide the URL. So what webpage you want to work with and what format the output is supposed to be. So if you say markdown then the scraped data is going to be in the markdown format. Now here is how the results look like when I tried to use the PDF file from Archive. This definitely doesn't look as great as what you get out of Jenna AI Readers API. There are a couple of other projects that I want to mention in this video and I'll be potentially creating dedicated videos on both of these. The first one is called Scrape Graph AI. So it not only combines web scraping, but it actually uses knowledge graphs to let you create rag applications based on the data that you scrape from web pages. They have provided a very detailed notebook on their GitHub link. But if there is interest I'll create a dedicated video on Scrape Graph AI. This project is under MIT license and they have pretty detailed documentations on their website. Now the second package that I want to mention is called crawl4ai. This is created by Uncle Code who also created some other really cool open source project and this one is available under Apache 2.0 license. Now this one is very interesting because it not only lets you scrape data from web pages, but you can also do a lot more thing things on top of that. So there are different chunking strategies that are implemented, there are different extraction strategies so you can do scraping and then build LLM applications powered by Rack and it also supports running JS scripts. So that's another powerful feature. And I'll be creating more detailed videos on both these packages. So these tools should give you a pretty good list of tools that you can start using in your own projects. Now usually the next step is going to be to build Rag applications on the data that you scrape. If you are interested in learning more about that, I created a course dedicated to Rag Beyond Basics. Link is going to be in the video description and one of the main focus of my channel is going to be to experiment with tools that you can use to build LLM applications. I will be still covering LLM news new models, but our focus is going to be on more practical applications and tools. So if that's something which interests you, I'll highly recommend to subscribe to the channel. Also make sure to enable notifications so you never miss another video. I hope you found this video useful. Thanks for watching and as always see you in the next one.