Marker: This Open-Source Tool will make your PDFs LLM Ready

The availability of good data can really make or break your LLM application. Most of the text data is available in PDF formats. This is true both for enterprises as well as for personal documents. But working with PDFs for LLM is extremely hard. PDFs is essentially a broken format. PDFs usually have complex structure. There are nested elements of different data types. There is absolutely no standard layout, which becomes very cumbersome to extract data from PDFs. There are added challenges because of different encodings. Different fonts, formatting tables and images can add to this headache. Now, to make PDF LLM ready, there are a number of approaches that people have been exploring. So there are approaches to convert PDFs to plain text for easier parsing. Then use machine learning models to detect the layout of your PDF. Then use optical character recognition or OCR models to detect text on the PDFs. Right? So it's actually a cumbersome task which is prone to errors. However, working with markdowns is very easy when it comes to LLM because you can easily convert that to plain text. Markdowns can retain the original formatting, so you can have title headers, images, tables in there. And LLMs can essentially effectively process markdowns structured elements. So the goal of this video is to show you an open source tool that you can use to convert your complex PDF files into well structured markdowns. If you want to convert your PDF files into markdowns, you have some paid options like Mathplics, which will convert your PDFs into markdowns or extract readable text from it. If you're looking for open source options, then you have NuGet, which is an open source project from Meta, but this is mainly focused on academic documents. I personally find this new project called Marker to be extremely helpful. This will Let you convert PDFs to markdowns quickly and accurately. And here is how it performs compared to NuGet. So it's much faster. So one page of text will take about 100 seconds compared to about 400 seconds when you try to do that using NuGet, and the accuracy is almost double that of Nougat as well. So here's a quick example of how the difference performance of NuGet and marker can look like. So this is a book called Think Python and the authors actually tried to convert this using both Nougat as well as the Marker. So here's the output that you get from NuGet. It basically completely ignored the first few pages along with the table of content. But Marker is able to preserve everything. So Here are the first couple of pages and then at the end we have the table of content which was accurately extracted. And after that we have the first chapter. In case of NuGet, it actually confuses the headers and footers as a part of the first chapter and also brings in the preface right in the middle of first chapter. But for this specific document, Marker seems to be able to preserve the structure of the book. So let's look at some features of Marker before I show you how to get started with this. So it supports a wide variety of documents, it's optimized for books and scientific papers, but I have tested in on something like resumes and it works pretty, pretty great. Now it says supports all languages. I'm not sure what exactly the authors mean by all languages. It removes headers, footers and other artifacts, which is evident when you look at that think Python book. And it formats tables and code blocks, extracts and saves images along with the markdown. So it will actually extract images and store them separately. And it will convert most equations to LaTeX depending on how the complex equations are. And the great thing is that it runs on gpu, CPU or MPS if you have Apple Silicon. Now if needed, it will do OCR on your text as well. It uses Surya, which is another package created by the same person who created Marker. Now there are some limitations as well because PDFs is a tricky format, so Marker will not convert hundred percent of equations to latex, which is understandable. And tables are not always formatted hundred percent correctly. I have actually noticed this. I'll show you a couple of examples of where it fails. So you want to pay close attention to that. White spaces are not always respected. Not all line spans will be joined properly. Now there are definitely some limitations, but in my tests it seems to work on most PDF files and it's an open source tool. But there are some limitations around the usage. So if your organization is making under $5 million in gross revenue in the most recent 12 months period and under $5 million in lifetime VC funding, then you can use this in your commercial projects. But if it's more than that, then you need to get a license, which is completely understandable because these open source projects actually takes effort times and compute cost to be done. Okay, so enough talking. Let's let me show you how you can start installing this and start converting your PDF files into structured markdowns. All right, so first we'll create a new CONDA environment and I'm going to call this Marker. I already have a version Environment by the same name. So it's going to ask me whether I want to remove the existing environment. So I'm going to say yes. This will basically delete the existing environment and then recreate the new virtual environment for me. Now, one thing you want to do is when you create a new virtual environment, you also want to install Pytorch. Okay, so we have our new virtual environment created. I am going to activate it using CONDA activate marker. And as I said, you want to install Pytorch. So depending on your operating system, put a link to this page, you need to select your operating system. I am currently using it on a Mac. So we're going to be installing using PIP and we have Python. So this is the command that I'm going to be using. If you are on Linux, here's the command that you need to use on Windows, Here is the command. But the only difference is going to be I'm not going to be using pip3, I will just use PIP directly. So PIP install Torch, Torch Vision, Torch Audio. And this will download Pytorch on my machine and install it. Okay, so install marker. We're going to use the PIP install marker PDF command. If you want to do OCR on top of it, you can install OCR MyPDF. It's an optional package, but here are the instructions on how to do that. In my case, I'm not gonna install it. I think it will use the Surya package by default. Okay, so once the package is installed, you, you can either convert a single PDF file to markdown or you can convert multiple files. In that case you are going to be using two different commands. So we're going to first start with converting single file. So you will use the markersingle command in your terminal. Then you need to provide the path of the file that you want to convert, then where you want to store that. And there are some optional parameters, for example, batch, multiplier, how many maximum pages you want to convert. And since it supports multiple languages, so you can actually provide the language which the document is in. This will help improve the OCR process if it has to do. But all the documents that I'm testing are in English, so I'm not going to try this. So for my experiments, we're going to be using three different files. Here is a scientific paper. These are basically two columns and there are some images in there. Now this is a relatively simple document because there are just two columns. The tables are well structured, so nothing weird in these tables. And there are some Images as well. And we also have some equations in latex format. So let me show you how to convert this PDF file into a structured markdown. Okay, so here my file is in this PDF files folder. So we're going to use the markersingle command, then path of the file and then the output folder. So I created an output folder which is currently empty. So it's going to create the markdown and store it there. Now when we run this, it will first download the OCR model which is Surya, if it's not already downloaded, and then it will try to figure out the layout of your PDF file. So it runs bounding boxes through the PDF files to detect different elements within the file. And that's how it's going to extract text and also ensure that the text is in the proper position. Right. So basically it first tries to figure out the bounding boxes and as I said, the structure of the whole document. And when it does that, after that it will run the bounding boxes algorithm on it and extract text from it. This will take a couple of minutes depending on the length of your document. So it took a few minutes and it created a new folder inside the output folder. So let's see what it created. So inside the folder we actually have all the images that are extracted from the document. So this is pretty nice. There are actually five images in the original document and it's able to extract all of them. So this is pretty neat. Now next we have JSON file with all the metadata. So it detected English language file type is PDF. There are a total of 10 pages which is accurate. And then I think it looks at the number of tables, the number of equations. So it was able to parse seven different equations and there are four different tables. So you also get this pretty neat data JSON file and you can parse this and see if the information in here is actually correct. Now here is the actual markdown that you get. So here's the title of the page, then the authors as well as their affiliation. So it also correctly extracted the abstract as well. And it all overall it looks pretty neat. Now for images, it's basically providing a reference of where the image is stored. So this is pretty neat. And I think it is also able to extract the equations as well. So that's pretty good. Now the tables in this case are nicely preserved. We can actually look at a preview of this. So I think if I right click on this and click on Open preview. So here is a preview of the paper in the markdown format which Looks pretty well structured compared to the original document. Okay, so here's the original document and I have the preview open on the side. Now I think it's able to preserve the relative locations. So here's the related work. We have the related work on top and then that image right in the middle. Let me have a look at the equation. So yeah, the equation seems to be correct. So this is pretty awesome. I have another document which is the CV of Andrew J. Yank, who is the creator of Coursera. So this is his CV and I ran this whole folder through marker. There's another paper called Orca. Now in this case, the paper has a table of content and then it's arranged in a single column and here are the outputs for both of them. So the CV again has a total of 21 pages. There are no tables or equations in there, no images. And here's what the output kind of looks like from Surya, which is pretty great. Here's a preview of the same thing. So here is the JSON file of the Orca paper. Now, challenges with existing models. Key contributions. I think these are different sections that the model is able to extract from the paper, which is pretty neat. One thing which I noticed was that it broke this image into two parts. So you might want to have like a secondary post processing step on top of this markdown to make sure that the images as well as the table that it extracts are accurate. But as a first pass, this is pretty awesome that you can do this with an open source model. Now here's the markdown of the original Llama 2 paper and it does a pretty decent job here as well. So this was a quick video on how to get started with Marker, which will help you convert your PDF files into structured markdown. It's an open source project, pretty amazing and it does a pretty decent job. And as we all know, data is an important aspect of when it comes to LLMs. So I'll be creating some more videos around it. I'm planning on creating a video on how to scrape data from web pages. So if you're interested in these topics, make sure to subscribe to the channel so you don't miss any of the new videos. I hope you found this video useful. Thanks for watching and as always, see you in the next one.