How To Choose The Right OCR Model

All right, welcome everyone. Today we have Joe Barrow. Joe Barrow is an NLP machine learning researcher. He's worked at places like Amazon, Adobe. He currently works at Pattern Data on NLP as well. And also there's a whole ecosystem of open models in addition to the paid APIs, and it's really worth paying attention to. And Joe has gone really deep in this area, and it's actually really helpful if you're working on any kind of AI product, because if you have documents of any kind, this is something you want to pay attention to.

Thank you so much. So yeah, the talk is how to choose an OCR model. I'm trying to make document processing as easy as possible, but that doesn't necessarily mean it will be easy. The first thing that I want to do is I want to convince you that you should actually care about OCR. Like Hamel said, a lot of people think it's sort of a solved task or that it's not particularly important to care about the nuances or details, but I beg to differ. The first reason is that your users care about OCR. So it wasn't until about a year ago that Claude could actually handle PDFs that were not just text. So up until about a year ago, Anthropic was just pulling the text out of a PDF. And if you uploaded a scan, couldn't see it at all. So this is like a large trillion-dollar company in product that is not handling OCR, and the users kind of noticed and complained a lot about Anthropic's poor PDF handling. Another reason you should care about OCR and not just pulling text out of a PDF using something like PyPDFium is that documents are heterogeneous. So for a recent project, we were OCRing, or we were processing like 10 million pages-ish of local laws to try and get all the data out, and a lot of them were scans or multi-column layouts where reading order is not necessarily present in the PDF. So if you use like a good OCR model, you're going to get homogeneous version of your heterogeneous documents. Another reason is that documents themselves are pretty evil, as especially PDFs, there's this really nice website from Vic Paratury from Datalab called WTF PDF, and you can upload any PDF and determine if it hits any of the really bad edge cases. So for instance, something that people don't necessarily know is that tech-compiled PDFs actually don't have spaces in them. Uh, it uses a system called glue where it's basically just printing glyphs at specific locations. If you just pull out all the text from a PDF without necessarily having some heuristics to post-process it, you're going to get just an entire run of characters with no spaces in it. And then another reason that you should care if you're building any kind of AI-powered app is that your whole is downstream of your OCR quality. So if you have your documents, the first thing you're going to do is try and extract all the information out of your documents before you do the actual task. Like, this is the business logic of the app, and then the thing that you display to the users. And what this means is that the OCR quality is actually a bottleneck for your app quality, um, and for your ability to do all this business logic on top of the documents. And once you've built your whole app on top of some specific OCR model, you have basically locked in to a specific vendor. OCR is very sticky. This is something that we learned at Pattern, basically, when everything was built on a specific OCR model or a specific like set of OCR models, it's very challenging to then go back and say, hey, I want to rip out this OCR model and replace it with something that's better or can do like better layout analysis or something. But what can you do about it? That's the whole purpose of this talk. And so a little bit about me. NLP PhD from Maryland. I led ML at Pattern Data until recently. I'm now actually at Adobe Research in the Document Intelligence Lab. But this talk is part of my— I'm here as sort of an independent researcher. So none of all of these opinions are my own and none of them are cosigned by Adobe. So the first question is, what do we mean by OCR? OCR, like I said, people think it's a solved problem. It's just about getting the text out of a PDF. But there's a lot of different things that OCR can mean. So if we take this paper on the left, there are 4 different versions of what OCR looks like. So the first one is just getting all of the text runs out of a PDF and, you know, concatenating them with spaces. And this was a pretty common approach to building AI-powered document apps for a while where you would just pull the text out and concatenate it with spaces and throw it in the LLM context and just hope that something good happened. And this actually leads to a lot of like quality issues with an LLM. And so you might imagine that you want to get something more rich out of your PDF. So there are OCR models that will extract, well, generate markdown for the PDF. In this case, we have headers, we have spacing by paragraphs, we have subheadings, and then models can also do things like table parsing. But sometimes you also want to be able to ground your model app or your AI app in the actual locations of the PDF. And so to you, OCR might mean I want to get the word or the line bounding boxes, um, so I can highlight that line and say, hey, this is where we think the relevant evidence comes from for this thing. And then for some people, OCR might also mean I want to get everything. I want to get the structure of the document and reading order, the layout, and all the semantic roles. So this is kind of a richer representation of the markdown where actually getting the bounding boxes of everything as well as the semantic roles and maybe some like rotation information and maybe a, you know, a table parse out. And the answer is that all of these things are OCR. So it's not just getting the text out of a PDF and it's not just getting the layout of the PDF. These are all things that we talk about or we think about when we talk about OCR. But the one you choose depends on what your need is for your product. So this is the broad landscape, I think, OCR models that are, is going to guide kind of the rest of this talk. I think there's sort of two axes that you need to consider when you're like choosing how to build OCR into your product. So the first one is, do you need just the text or the text blocks out of the PDF and you don't care about the semantic roles like headings or paragraphs? Or do you need the full structure? Do you need the tables out and the figures and everything else? And that's going to determine, I think, a lot of things including cost and quality. And then you can attenuate those by deciding if you want to try and self-host a model, which can, you know, lead to lower costs but more engineering time, or if you're going to rely on an API. And if you split out these two axes, they sort of, I think, neatly cluster into four groups of models that you can consider. So we have big cloud providers like Amazon with AWS Textract or Google with with Google Cloud Vision. We have a whole slew of new startups that are building what I think of as sort of like fully featured OCR models. So these are things like Reducto, DataLab, Extend, LlamaIndex. And then on the self-host side, if we're just getting text blocks out and we want something cheap and fast, we have open pipelines using things like PaddleOCR, NemotronOCR, and Tesseract. And then if you are going for open VLMs, something that can do more of the structure, then we're looking at models more like LightOnOCR, GLMOCR, Chandra, Chandra2, Docling, SuryaOCR. Uh, in this case, there's like probably 50 to 60 open models, but probably only a few that you might actually want to consider. So how do you choose which quadrant to be in? And in a quadrant, how do you choose which model to decide on? The first question is, do you need text blocks or do you need structure for your downstream app? So there's a reason that you might choose text blocks if you wanted to ground your LLM output to a specific line or a specific, like, set of words, because text blocks from APIs can give you word bounding boxes, versus do you want structure, which can actually give you a much better input to a downstream LLM because you're actually getting, like, a sort of semantic representation of the document in reading order. This document structure gives you things like grouped text. So whereas text lines just give you each individual line or block, grouped text will say these are all part of a paragraph, these all sort of belong together. The semantic role, so the paragraph, the heading, and the table. Reading order, which is something that you're not necessarily guaranteed if you use the line-level OCR. And then there are some models that will also give you layout. So this is getting the actual bounding boxes of each of the elements. And then things like parsed tables, figure alt text. And I think really interestingly, more recently, things like chart de-rendering. So if you actually have a chart, it will de-render it into a, effectively a CSV of the data that was used to construct that chart. But these are things that are not present in every model. These are things basically that when you choose a model, you might get access to this. Basically, you would choose text blocks when you need grounding to align a word, if your app actually needs to, like I said, show evidence, and typically if you want the cheap, cheapest option. So if you want the cheapest option, it's almost an order of magnitude cheaper to go for a model that just does text blocks because they're typically simple pipeline models. And that is true of the cloud APIs versus the startups as well. And we'll look at that a little bit later. You would choose structure when you typically want the cleanest option and you want to preserve the visual semantics of the document. So in this case, case you want to actually know where the paragraphs are, or basically what the content like actually looks like in the document, and getting that out into, say, your LLM context. And this actually does matter. So this is something that we noticed at Pattern. It's basically a free performance improvement for your downstream like information extraction or AI-powered question answering task or whatever if you actually preserve the semantics of the document. So if you're just giving it runs of text that you get from Textract because that's the cheapest and fastest option, you're typically leaving performance on the table. And so if you want to build like some RAG pipeline or information extraction pipeline, which is a lot of what people using OCR and AI together nowadays for, uh, you typically want to go for something that can preserve the structure of the document. And a lot of this is because like LLMs are trained with a lot of markdown in the training data, or a lot of like similar things. And complex document layouts, uh, when you just give them lines, are going to basically give the LLM like jumbled up garbage, including from like the really good providers like Textract. And then when deciding whether to pay for an API or self-host, basically with an API you're paying for something like ease of use and support. So something that's really nice when you talk to one of the startups, basically if you have documents documents that are like not working well with their APIs, you can typically talk to them and say, hey, I have this set of documents and I'm getting bad output from them, and they will consider that when they're retraining their models and their pipelines, especially if you're like a larger enterprise customer. And so that's something that you couldn't get if you self-hosted the model. You basically have to figure out how to fine-tune it yourself. And you also want to use an API when you're trying to put in minimal time and effort to actually handle the OCR for your app. Not everybody has the time to build out an OCR pipeline. I think it's not as complicated as you might think and is definitely doable to for scaling, but it is a time-consuming thing when you're actually focused on building out a good product. And you'd want to choose to self-host when you want to control things like throughput or concurrency. So typically when you go through an API, you are limited to some fixed concurrency. And we noticed this at Pattern that we were often bottlenecked by the number of simultaneous documents we could have in flight with our OCR providers. Whereas hosting something on Modal or in other infra, your throughput is something that you can kind of pay more for typically, as opposed to having to talk to a team to try and get more throughput out. You also control the weights your OCI model OCR model will not change underneath you, and you get control through the ability to fine-tune for in-domain or add, you know, speculative decoding or something. Self-hosting is also cheaper. This math generally only works out if you value your time at $0, but it can be a lot cheaper if you're considering hosting a VLM-based OCR model versus using one of the more expensive, like, Startup APIs where you're paying something like $5 or $10 per 1,000 pages. You can get that number kind of way down. It's also cheaper if you are running in, like, substantial bulk. So if you're running, like, a huge batch process job and you're an enterprise or something,, then the math can work out that even if your time isn't $0, uh, but if you're building, like if you're prototyping an AI thing and you're trying to find product market fit, it is not going to end up being cheaper, but you avoid getting the vendor lock-in. And then the other reason to self-host is just that it is incredibly fun. I think it's like a really neat engineering challenge to solve both the, like, how can I do batch OCR and how can I do sort of online OCR with like high throughput? And it's the kind of thing that you could get like a nice first pass with like a weekend worth of effort, I think. But obviously building like a robust system that can handle like large throughput is, is not just a weekend worth of effort. So that's kind of comparing the two axes, but I also want to dig into each of these different quadrants and then how you might consider to choose between them. So if you're choosing between like going for a big cloud or going for startups, the big clouds like AWS Textract are typically something like $0.60 to $1.50 per 1,000 pages. So Textract starts at $1.50 per 1,000 pages, I think. I think so does Google Cloud and Azure, so it's like a dollar per thousand pages. And then there's volume-based pricing after that. So after your first million, 10 or 100 million pages in a month, the price starts to drop and Textract bottoms out at 60 cents per thousand pages every million pages. But typically this is just giving you word and line boxes with their text. So you're not getting that semantic structure. If you want to get table parsing, you're going to have to do that downstream on your own. If you want to get groupings or clusterings and reading order and like reliable reading order, these are things that you might have to consider doing on your own. And if you want these things, you can cut— like AWS has a way to pay for it. But you're paying à la carte. So for instance, if you want form processing or table processing, you're paying $10 per 1,000 pages or $15 per 1,000 pages for each of table and form processing. Basically, the alternative is to consider startups where— so things are things like Reducto Extend, DataLab, Llama Index, where you're paying something like $5 to $20 per 1,000 pages. And the distinction there— there is volume-based pricing discounts, but the distinction there is that the lower end you're paying for what they typically call something like fast, and at the upper end you're typically paying for something that they call like accurate. Each company basically has the distinction, but they call it something different, but you can think of them as sort of like fast and accurate. But what you're getting out of this is you're getting table processing and figure bounding boxes and grouping. So like, you know, the actual markdown or HTML of the page, which can be much more useful downstream when you're feeding that into an LLM. And then on the open source side of things, you're comparing and contrasting pipelines and VLMs. Pipelines are typically— so these are things like Tesseract and PaddleOCR. They're really fast and cheap to run. They're between 10 and 100 million parameters typically. PaddleOCR has a version that can be run from your phone. And in fact, there are a lot of devices that just run PaddleOCR locally, including if you have like a Boox device for where you can, you know, write on it, kind of like Remarkable. These are just typically running a model kind of like PaddleOCR, but they're getting typically text lines only, though there are some like Paddle recently released DocLayout model for layout, and then you can kind of use that to post-hoc aggregate text lines into paragraphs and such. Conversely, VLMs are much heavier. They're normally on the order of like 600 million to about 8 billion parameters. And so you can imagine that cost is commensurate. Sort of model size. Document structure is natively passed out of the VLMs. So if you look at something like a Chandra or Surya, you're getting the blocks out, I mean the document structure out, as opposed to like just the text lines. There is a belief that they come with more risk of hallucinations, and that, that can be true. So for instance, for certain open models, if you pass in an empty page, like you, you risk getting some sort of like boilerplate hallucination out. But in my experience, looking at like at this point probably a million pages of OCR output from like the big clouds as well, you run that risk also with like really crusty scans. When you pass something through Textract where they're doing the text line recognition and then, or the text line detection and text recognition on top of that. I've seen things like just the word "the" repeated 100 times when it's like just effectively like curling noise in a scan. And then with VLMs, you're also paying at the compute rate. So when you're running something like 600 million to a billion parameter model, that's something like 20 cents per 1,000 pages on an H100 for every, like, assuming you're saturating the H100 at all times. And up to an 8 billion parameter model where you might be looking at something like $3 or more per 1,000 pages to run on H100. The open pipelines, like I said, they're typically composed of many small models. So this is the Paddle OCR set of models where they do page orientation detection to determine if it's rotated. They dewarp if it's a scan. They identify where all the lines are on the page, rotate those lines if, if they're separately rotated. Like on the Archive paper, there's a header on the left and then different orientation, a line on the top, and then text recognition of each of those identified lines. And that's why with these open pipelines, you're typically just getting the line bounding boxes as well as the text within each individual line, and not necessarily the reading order or the document structure. Conversely, VLMs are typically just one big honkin' model trained end-to-end where you feed in the page and the prompt and you get the output text of output text is maybe some HTML format with bounding boxes. That's what Chandra and Surya do. It's like the Quen HTML format. And it's just a single shot typically. There are multi-shot or like pipeline VLMs that are kind of outside the scope of this talk. And I will like one word of caution, if you like over-index on cost and latency and go like totally for pipeline models and you're using handwriting and like you're not looking at the quality, the models can be really fast, but your entire app is kind of downstream of your OCR quality. And so it's worth considering paying more. This is the sort of like image that I pass around for this idea. You can process 1000 pages per second, but it doesn't matter if they're all like wrong because then your entire app like output's gonna be garbage. But really again, when choosing like what model, what quadrant to be in and like within that quadrant, what model, it really depends on what your actual need is. So I wanna go over two use cases that one of them is a real use case I have. The other one is like stand-in for a use case I had. So the first one is this dataset Laws. So like I said, we collected about 7 million pages of laws from around the United States from every city and county we could get our hands on, which ended up being about 7 million pages of, of PDFs. And this was a project for, with a friend at Berkeley. And so this needed to be done on the cheap because we are not given like huge resources to be able to process all 7 million pages. You can imagine at $5 per 1,000 pages, that's more than the Berkeley lab is willing to pay. And this is like a hobby for me. So it's not like something that I'm willing to put a bunch of money into. But what we really need out of this data, out of these PDFs, is just the text, the headers, and the reading order. So we can split the data into each individual law and then make sure that we have things like coherent. So if we have a 2 or 3 column page, we want to make sure that all the laws are grouped and coherent, but we don't need graphics. Rounding figures and tables because we don't actually care. Like, we're gonna throw away a raw PDF at the end of the day anyways because we're just trying to find the laws and release that dataset. And then my time in this case is valued at $0 because again, this is a hobby. So the option is to self-host a VLM model that just outputs Markdown. And in this case, it's LightonOCR2 from Lighton AI, which is a really strong 1B model that just outputs Markdown and is pretty close to the top of one of the benchmarks, AlmostCR Bench. And I think that in our case, this was the right decision. We could basically run this about 30 cents per 1,000 pages was our final number, all in, including reprocessing and stuff. And it ran quickly or effectively over the course of a weekend or a week. Conversely, if we consider this, let's say, startup where you have some vertical and you want to do information extraction for that vertical, so you're doing this for legal companies or for like hospitals or pulling data out of financial records, and there's some live user who is actually like looking at the, like uploading PDFs to your system and asking it questions or getting some information extracted, you're going to have kind of a different set of constraints. And so this is a pretty common, I think, like startup setup in the space where you're building on top of an existing OCR model, but you're adding a bunch of value by using AI to do information extraction and like business logic. In this case, you don't really have a dataset. You're not doing like batch data. You're handling it live. And so you kind of want to be able to scale to, you know, you have a lot of users. If you're doing grounding to the actual PDF, as we are going to say we're doing in this case, you're going to need the word bounding boxes because you want to be able to highlight the individual words that like are relevant for your AI output. Let's say that you, if you're doing this, you might not need structure. Maybe you do want structure to feed your LLM. So you have to build some sort of post-processing system, but in this case, I'm working in my startup. My time is very much not free. Basically, there's an opportunity cost to spending too much time building, like working on my OCR and tinkering with that, as opposed to building the app and the business logic and the thing that provides value for the downstream customer. In this case, a good decision might be something like Textract and then build maybe some post-processing layer on top of that to aggregate things into structure and like feed that to the LLM. But again, I want to kind of underscore that there is no best. It really depends on what your actual need is. And you have to determine that by looking at outputs of different approaches on your data. So how do you decide whether or not it's worth self-hosting or using an API? I would say that most of you probably should not self-host and should not want to self-host. I think probably less than 5% of you, I think is a fair estimate. If in doubt, if you're really product focused, if you're trying to find product market fit, go for an API. It's not worth your time. Like you are not the OCR person. You're trying to like be startup for financial records person. But if you are in that 5%, I will say that hosting is easier and fun as hell. And so like most of the rest of this talk is for you. So how do you choose which model to host if you want to host? First, you have to look at the actual data that you have, right? Do you have really crusty scans? Do you have handwriting? Is there a bunch of medical terminology in it? You're basically going to want to look at your data, and you're going to want to ignore all the benchmarks. There are some, like, really popular OCR benchmarks like OmniDocBench and OlmoCR, and those are not going to tell you how these models perform on your data. They're sort of indicative. At the end of the day, like, they're not run on your data. So the first thing I want to underscore is that the only way to choose an OpenOCR model is to test it on your data. So how do you do that? First, you're going to manually build a set 50 to 100 representative pages. I think it's a good size. And the easiest way to do this, and I know this sounds really stupid, but open a bunch of your PDFs that you have and copy and paste them into like one new PDF that you are creating with like Apple Preview or Adobe Acrobat or something. If you have them all in one PDF, it's going to make it easier to just like either hit the different cloud APIs or like hit your hosted server. Then you're going to choose a few candidate models basically from that grid, and you're going to run a few— run that like 50 to 100 page PDF. And then, and again, I'm just I'm just gonna like keep stressing this. You're going to look at the data. So these, the way to do this, you're gonna diff return text between these candidates and see if one, if like diffing return text is an easy way to see like, oh, this model's returning junk on handwriting, which is something that a lot of like pipeline models do. You're going to visualize boxes. In this case, like this is something that you can kind of easily, pretty easily like Claude up basically as like visualizer for your OCR outputs. And you're just gonna spend some time looking at the data. It's like 50 to 100 pages over, you know, 5 to 10 candidates. Ideally, all of this is about a day's effort of you just like looking at a bunch of OCR outputs and, you know, standing up some like VLLM instances. Another thing to really consider though, and I really want to stress this, is look at the license of the model though before you choose the candidates. So for instance, Chandra and Surya from Datalab are really good models, but they're only free if your org makes less than $2 million a year and you're not competing with Datalab, or you've raised less than $2 million. I think you have to have both. Conversely, there are really good open licenses that you can use at any stage of commercialization. So for instance, LightOn OCR is Apache licensed. GLM-OCR is MIT, so they rely on PaddlePaddle's doc layout model, which is Apache license. So you actually kind of have to adhere to both licenses. But at the end of the day, like, do not like start a poor company at risk by using a model that you don't actually have the license for. Plus it's kind of like good karma to adhere to software licenses. So how do you actually host the models? I would say that you need to do 3 things. The one is you need to have decided on a model. Two is that you need to decide on some inference engine to run. I really like VLLM, and a lot of OCR models are released with VLLM as like sort of the default system to run on, though SG Lang is also very good. And I would about half the models that have been released and are like tied to an inference engine have SGLAG support as well. And then you need to choose some infrastructure to run that model on. And I really have enjoyed using Modal. Moved everything at Pattern over to Modal. And for the like Locus stuff, we basically built the OCR on Modal. There are other really good infrastructure providers as well and inference providers like Base10. And then you can always use like cloud, like big clouds like AWS, though the shape of your needs are going to kind of determine what you want. AWS is great. I think if you have like a huge batch of data you want to process once, but SageMaker scale in, scale out is like kind of clunky compared modal, just like easy scaling based on the number of requests that are kind of in the queue. And then basically from there, you can get surprisingly far by just running one of these two commands in a modal container. These are both about a billion parameter models because you're running them through VLLM. You're exposing like an OpenAI stock client and you just need to pass in the GLM OCR prompt for JLo OCR. For Latent OCR, there is no prompt. You just pass in effectively the image as like a user message. And both of these models are running on an H100, assuming you're saturating it in like batch. In my experience, typically around $0.20 to $0.30 per 1,000 pages, even at sort of modal prices. And you're getting something like like 10,000 pages an hour on an H100. And I think I did, it's like 3 pages a second and 3,600 seconds on an hour. So something like that. And I found, I've also served LightOnOCR on like my local rig of 3090s and on 4 3090s, you're also getting close to like 3 to 4 pages a second, assuming you can saturate the GPUs. 4 3090s is equal to 1 H100. But going back to the decision. So that's kind of how to host. Like I said, most of you probably shouldn't consider hosting. It is fun. If you have questions, please reach out and I would love to talk to you about it. But I guess back to the decision matrix, if there's like one takeaway from this talk, I hope it's kind of this. You want to decide which of these two axes, like which quadrant from these two axes you want to land in. And then my annotations would be for full-structured document startups, you're paying a lot of money, but you're getting like high-quality and sort of feature-complete outputs, and you can do like stuff those into an LLM. For open VLMs, they're typically heavyweight, they're roughly feature-complete, they're typically a little bit worse than the document startups. A lot of the open VLMs are released by startups like Chandra and Surya are from Datalab, and they have an even better in-house model. But again, it's like cool and fun as heck to do, and so, you know, do it for fun if nothing else. For the open pipelines, you're talking something like Paddle OCR, which can be very high quality, but still limited compared to everything, all the niceties you get from OpenVLMs, but they are fast and very cheap, even running, like capable of being running like on an edge device, which would be free for you. And then with big cloud providers, you are typically running like at scale, typically quite cheap. You're getting like word and line level bounding boxes out. And that brings me to the end of what I have prepped. Don't hesitate to reach out. You can hit me up on Twitter @BaroJoseph, and I am like always happy to answer questions.

Amazing talk.