How To Make PDFs Your AI Can Actually Read

Datalab came up with this accessible PDF API that takes PDFs and turns it into stuff for people who are blind to be able to read, 'cause a lot of that is, you know, really not accessible at all. So let's take a look at the post. Okay, so it's a new processing pipeline for accessible PDFs. I don't think they claim that this is everything you need to be accessible. They still recommend human in the loop, but let's see what we have. What has to be tagged? It's a set of criteria about Whether a person with a disability can perceive, operate, and understand a piece of content. Does someone reading with a screen reader, a Braille display, or reflowed text get the same content in the same order with the same ability to move it around as someone reading a page with their eyes? So this seems really important to me.

One thing that, that we need to talk about is what the hell do we mean by accessible PDF? Most people aren't blind. They have no experience with Accessible PDFs. So like normally when you use a PDF and you try to get a screen reader to read it to you, let's say you're blind, it'll just garble everything. It'll like say it in the wrong order. Let's say the PDF has multiple columns or has a table, it'll just say it in the wrong order and it won't make sense to the person hearing it, which is like a huge problem. Most OCR is bad at this accessibility thing, and I think that's what they're talking about.

This first thing they show is a document And the structure of the document, I guess. So headers, paragraphs, lists, formulas. Okay, so alt text is in pixels. Figures are cropped and described based on what they show. So we need to take graphs and furniture and stamps and describe them. Formulas need to be spoken like LaTeX. If a list is broken by page break, that seems page furniture, printed form fields. Vic, who's the founder of DataLab, shared this for accessible PDFs. If you're blind, most PDFs are a garbled mess. And having one fixed can cost over $100. Their API makes them usable for pennies. And he's got this video. Let's take a look. Okay. So there's integrals and formulas on one side and the page's text layer spoken in order. Prove that. And then a screen reader receives it in a little bit more structure, text and formulas.

So there's a lot going on here. Can we talk about what happens by default in screen readers with PDFs? And like, why is it so expensive? Like, let's give some background on this.

With a PDF like this integral calculus one here, we'll just use that as an example. That's most of the time just going to be like a scan or an image. It's just like you would expect from like a JPEG or a PNG file where there's really nothing there. Now, sometimes there's like a text layer behind the PDF, but a lot of PDFs don't have it, especially for if you've gotten downloaded a PDF of a book or it's got a chart or it's got all kinds of stuff. And so a screen reader might get nothing, or it might use like a really light OCR model to try and like OCR it. But if there's no structure, then that's kind of— it needs to identify like what is a title and what are all these different things in order to, you know, read them differently. Okay, so like the text layer here, if we take this integral calculus and we were to read it using the page's text layer, which might be the default: 11 period prove that 2 question mark a a plus b square bracket Plus asterisk DX. So it's like completely—

Yes.

Reading that out is completely useless.

It's the text layer that is messed up in this PDF. And that's what he's saying is like, you have to go through and painstakingly fix that, especially for complicated things. Like you have to go like do some surgery on this. Is that right?

Yeah. I mean, in a lot of cases the text layer doesn't even exist. So, you know, in this case, I don't know, maybe you could pass it into an LLM and have it just take the text layer and correct it possibly. But in a lot of cases, even that's impossible 'cause there's no text layer at all.

Yeah, and you have to have some really great OCR to like fix all this stuff. Like you have to know what's going on and do it accurately. So what is he showing here? What's this screen?

So this is the same integral calculus, but a screen reader gets it in much more structure. So you see text, 11, prove that. And so you can say, well, this is just text in prose, and you can handle that for how it's going to read that. And then it says formula, and it gives the whole LaTeX formula. So then that's structured again. So Maybe you don't want the formula just read out, or maybe you do, but you can have special handling for how you want formulas handled. And then you have different text layers and formulas. And so it breaks it up in a way that you can handle different things. It's actually usable. And that's one of the big things that is a challenge for screen readers, not just for PDFs, but like for web as well, is that when we look at a page, I immediately see there's a lot of like visual information that's not just straight text. For example, I see Vic here. I immediately see that that is the name and that this is like a lighter gray and less important. I can see that like they're on the same line, but it's different pieces of information 'cause there's these like separators and like bold and highlighted. And then underneath I see these line breaks. And so even in something simple like this, I see a lot of structure represented. And if we go back here, It's like same thing in this integral calculus. We see that prove that, and then this is indented underneath. There's that structure that we know that means it's a formula, like at a scan. I don't have to read it to know what it is. And so we see a lot of this structure. Same with like page number. And then this, because this is centered, it's very obvious that this is a title and not part of the text.

Yes.

And so we visually see all that structure, and just getting text and jamming it all together Loses a lot of that.

And by the way, he's saying having one fixed can cost more than $100. He's referring to the fact that this is a math textbook. We take an entire math textbook, fixing everything, going through it meticulously, because in these environments, especially in education, you have to be accessible. You know, there are certain industries and certain situations like you have to be accessible and you should try to be accessible. And so these things can be really expensive to get right, even with AI. And I think that's what he's showing today is like he's tuned a model to like do this really well. Is that my understanding correct?

Yeah, yeah, absolutely. And I mean, the cool thing here is that like it sounds like it's this experimental thing, but it's not. DataLab did this a little bit ago where there's actually Davis County School District is using— is bringing STEM to Braille. And so there is a teacher that works with visual impairments in Utah, and he's a certified Braille transcriber, and he produces all the STEM materials himself because a lot of times like People come midway through the year or stuff isn't done through the school system. And so this has sped him up from like hours to minutes. And if you look and you compare what this looks like and all this structure that gets here, this is about what's happening here, right? Like you have this and you're putting it into structure. And so this API is accessible, but it's being used in real cases in education.

That's amazing. Okay, so what does he talk about next? He's like, we're not claiming to get 100% compliant. still need a person, just like you said, you might need someone, but maybe instead of hours, you could just take minutes.

Yeah. I mean, generally, I think when you're using any kind of AI or ML thing, human in the loop is really good, especially if you're doing something where, you know, mistakes can be harmful. Yeah. I give instructions for how to do it. There's a bigger blog post.

You have to wonder also, like, is this structure— it might be good if you're not blind. If you're trying to feed this to downstream LLM, I don't know, like maybe the structure is better.

Yeah, well, I think the structure could be useful because a lot of times models are blind, right? They don't see very well unless they pass in an image. And a lot of times these models, they pass in an image, it's like, what are they going to do? They're going to run OCR on it so that they can read it. Yeah, it could be good there. Datalab also has other models like Chandra that will give it different types of structure. Like their Chandra model can output HTML, or you can use Gemini to put out JSON. So depending on what you're doing, you might want different types of structure. But yeah, I mean, certainly here, a table versus a list versus text versus a footer, you might want to programmatically handle those in different ways.