Let's Convert a PDF Research Paper to a Podcast using AI Agents[Real Quick]

Time to get started. Let's see, should already be live on all the streams. Just want to confirm we should be live on Twitch and live on YouTube as well. So that's, that's still taking a moment. It's not live yet. So let's give it a moment and see if that's going to go live. And yes, seems like we're live on YouTube and now live on Twitch, but the Twitch one looks it looks a bit funny. So let me go ahead and make sure that that works as it should. Cool. I think we're good. We're game. So in today's stream, I'm just gonna go over. It's gonna be a really short stream, to be honest. It's probably gonna be even less than fifteen minutes. I want to go over and and build something. So one of my friends, Corner. Connor Shorten. He works at Weaviate. Connor, if you're watching this at some point, he sent me a paper that he wrote and I'm going to share this here as well. He co-authored a paper on— and he shared here. Let me put it this way so you can see it. So there you go. Hey everyone, I'm super excited to share on a research report on arXiv. So they released a paper on arXiv called Querying Databases with Function Calling. Which is pretty interesting. So the capabilities of large language models are rapidly accelerating, largely thanks to their integration with external tools. Querying databases is among the most effective of this integration. So using essentially what they're doing is I think using function calling to query databases, which is something that's pretty nascent and new at this point. So it's pretty interesting to see what they came up here with. So they looked at, they talked about OpenAI Deep Research, how it's been amazing, the design of effective compound AI systems. And this is just his tweet. I just converted it into a readable format. But he goes into detail here exactly what it is. He shared this with me. And what I want to do is, which is pretty good. He sent me this. I ended up I mean, previously I built an AI agent that would go ahead and get a given paper, for example, a PDF, and it would use AI agents to kind of creating embeddings of that paper, which is going to be in a PDF, create embeddings of it. And then I can generate summaries of that paper using AI agents as well. And the next step after that, of course, I could generate like a transcript of 2 people or dialogue of 2 people talking about that specific paper, for example. And I could use something like ElevenLabs. So if you're not familiar with ElevenLabs, it essentially allows you to have or generate voices using synthetic voices using AI. So this is the paper. I'm going to download it and I'm going to save it to my system. So I'm just going to put here, Corner PDF. That is saved. Now, if you go to my project here of AI agent that I built, I might just do like a quick overview of it. And this is on Colab. So I'll share the link as well. If anybody's interested to go play around with it, if you go to my repo, you can look me up, Tony Kipkemboi, on GitHub. And you'll be able to find this repo here as well. So it's the paper, research paper, the podcast generator. And so this is the flow. We'll put in the PDF that we have, which we just downloaded. And one of the AI agents is going to take that paper, look through it, generate a summary of that paper. And then it's going to pass that to another agent that is going to take the summary that the first agent has produced and generate a script of 2 people talking about the content of that paper, for example, in this case. And it should be very engaging and human-like. And I added another agent to kind of just look at the first script and try to make it more lively, make it more human, add more features to it. And then once that's done, the last agent kind of sends each, like each individual chunk or each individual dialogue for each one of the hosts, send that to ElevenLabs. ElevenLabs is going to generate an audio for each and every word or sentence or segment in this case. Of each of the speakers. And it's going to save it in a folder. And once it's done for all the dialogues, it's going to just going to compress all of them together. I'm going to use PyDub in this case. It's just going to do some good editing, put all the individual MP3 files together to form one full podcast. So let's go ahead and run this. So I already have set like my API keys here. So if you come here to the API key, I have a bunch of API keys on this side right here. So I've already saved it there. That's the best way to do it really instead of like saving it here. So let's see. So I'm going to start running this and it's going to start running, just downloading the packages for that. And then the next step would be of course to get those API keys that I have on there and get them ready. And just to go over the API keys here, let me just zoom in a little bit more. Cool. Hopefully that's helpful. API keys that I have is OpenAI. I'm going to be using OpenAI mostly. ElevenLabs API key. I added Anthropic and Cerebras and I had Groq as well and Serper service, which is searching the web. So that's something else that I missed to mention. So there's a capability for it. There's another agent that, for example, if that research paper that you pass to the agents, maybe that paper was written in 2014 and stuff has changed since 2014. I mean, if you've been awake for that long, stuff is different now. I have another agent that kind of does recent updated research on that specific topic that was mentioned in the paper and kind of adds that as context to the paper so that when the two, like when the two hosts are talking, they're able to infuse new information and they explicitly mention, based on recent research as well, blah, blah, blah, this and this happened. So that's just kind of like adding newer context to it. So in case that paper is very old, for example. So I'm just granting access to that. Grant access, grant access to all of them. Access, access, access. Boom. Should have access to all of them. So this, all of this are there. The one thing here that I added as well is like the voices from ElevenLabs. So if you go to ElevenLabs, you should be able to find your voices there. So if you go ElevenLabs, actually let's not go to the docs. I want to go to the main app. So if you go to the main app, you'll be able to see voices. So if you just log on, you might need a credit card though. So if you go to the voices here, uh, actually they— I think they have some free credits you could use, to be honest. Um, so if you log in, you'll be able to get free credits. If you go to the voices, this is my voices that I've already added, like that I've found. But if you go to the library, if you don't have any yet, if you go to the library, you'll find a ton of voices. What I recommend for podcast sounding voices would probably be if you go to here to conversational, you'll be able to see a bunch of conversational. So what I recommend is just clicking them, listen to them. You find one that you like, you add it to your voice. And how you add it is just basically come here, you click add. And once you've added, it's going to pop up on your voices. And the ones that we're going to use, for me, I chose Ben and Claudia. And so if you come here, you pick whatever you want. Maybe you want to pick Christina, for example. If you want to use it, you need the ID for it to use it within your agent or to use the API key. So you just copy the ID and me, I saved it here in my keys here. And then I'm able to call that into the file and loaded them up here as well. So the next thing is to build the tools and the tools that I built here were tools to get the, because agents, of course, they need tools to get the job done. So in this case, I built one for voice. So these are just like configuration to make this voice voices of this host sound much better. So that's what I did there. And I have this tool as well that kind of generates, sends each segment to the API, ElevenLabs API to generate the individual voices so that this is what this section does that. And I just ran that. And then we have mixer tool, podcast mixer. This is the one that now takes all those individual voices and puts all of them together to form one final cohesive podcast, a full MP3 voice there. And now this section here generates just the directories. So I want to save like segments, for example, those simple segments, I want to save them in a directory, also the transcript. So I can review afterwards to see what that looks like, like the script that the, or the dialogue in this case. So I'm going to run that. Now we get to the point of the fun part, which is uploading the PDF. So upload the PDF, choose the PDF, and that should be this recent one, Connor. Load that. That one is going to load. So Connor, if you're watching this, I'm going to convert your paper into a podcast so I can listen to it if I don't want to spend more time reading it. It's like a shortcut, but I definitely will read it again. I'll make sure I read it. So here you go. And we've uploaded the paper. Now we set up the agents. So here we go. First of all, we want to clear the memory. So if there's a memory already that it's that these agents are already saved, we want to clear that. That way we We start from a clean slate. And the next step now is to pass that PDF into the PDF knowledge source. And what this does really, this is a function that we have in CrewAI. This takes that PDF, converts it into vectors, creates, essentially puts it in a vector database. And under the hood, we use ChromaDB for vector databases. So that's something you can also play around with there. So that's what I added in there. Now the next step now is generate Pydantic models and Really pedantic models help streamline or have consistent outputs of what you want. So for example, this paper summary model here will look at the paper and it would extract specific things from that paper. So it's going to look and grab the title of it, make sure we have the title, get the main findings of that paper as well, methodology that was used in the paper, key implications, limitations, future work mentioned in the paper. And this summary data is not important. I could definitely do without this. But I just added that for the fun of it. And then there's a dialogue as well here. And this is where like the conversation, it's going to be a speaker, the name of the speaker, and then text that they're going to speak. And then we also have podcast script and this is just the full dialogue now. And then we have audio generation and this is the segment file. This is those individual stuff that each one of them says. And then the next person says those are segments. And then the final podcast is the one that's put together. Now after that, we got to configure the LLMs. This is like the brain of the agent now. So the brain of it here, I have the summary LLM. And the cool thing about this, I'm using CrewAI for the agents, by the way. Each agent can have a different LLM that they can use. So in this case, I'm using OpenAI O3 Mini for the summary to generate the summary of the paper just because it has more reasoning, of course. And then I'm also using here the script LLM to generate the initial script that I want. I'm using O1 to generate the initial script. And then I'm using O3 Mini to enhance that first script that's been generated to make it even better and more human sounding in a sense. And then the audio LLM, this is the one that generates the final audio, which everything is done at that point. I'm using GPT-4 Mini. I could use different models here. And then now let's go ahead and set up the podcast voice hosts. So the host, one of them is Julia. I named them Julia. The one I named Guido. Those are just random names. You can name them however you want. And I'm going to run that. Now the next step now is run the agents. I have agents. I have a researcher agent. Who's going to just do the analysis of the paper. And then I have another research support that's going to go to the web. Like I mentioned, it's going to grab the most latest information that relates to the topic mentioned in the paper. And it's going to augment that as the knowledge that we need to add to the paper so that when, when we're talking to, when our speakers are talking or our hosts are talking, they can reference more up-to-date information instead of giving us stale information. And we have the scriptwriter. It's going to start writing the script for the speakers and then the enhancer, of course, to enhance this, the first script. And then you have the audio generator that's going to generate the final audio together. Now the task, these are just basically stuff that get assigned to the agent. So think of this as more of like a job description. So this description would be like a job description. For example, if you're human working at the company, this would be a job description. And then the expected output is more of like what's expected out of you. So key performance indicators, things like that. And then you have the agent who's in charge of all this. And then output by Dantic is just what we add up there, what it needs to extract from the paper or what it needs to actually get from whatever they need to do. And then we have an output file where it's going to output the task that it did. I wanted to output to a file so I can review afterwards to make sure that it did it correctly. And that's just another clean way of reviewing the work of each of these agents and figuring out who did the wrong thing or who could be optimized to do a better job essentially. And now here I put the crew together. So I put my, all my agents together. I put all their tasks together, their process. They're all gonna run sequentially on the knowledge, which is the PDF. And yeah, just putting everybody together. Now I'm just gonna run that and it's gonna already start. There you go. So it's kicked off. Let's give it a moment and we can see the researcher is already at work. So it's going to do research. It just took the paper there. Function calling, evaluating search over structured. There you go. So it got the name of the paper itself now. So there you go. Function calling for natural language database access. Okay. All right. And we have the support specialist. It's just going to research the most latest information, which the paper is really the most recent. So this might actually corrupt a little bit, but I mean, it's good. You know, we get extra context of all the papers that are there because I think it's grabbing other papers as well. So we're going to use all those as references as well. So not bad. Let's see how long this is going to take. Shouldn't be long. But we should also see here that it's generated the outputs folder. And within outputs folder, we could see here. What time is that? It should. Find the most latest one. All right. What's the most recent one? Okay. So here is the podcast script, right? The first one is generating a script right now, but you can see it's saving the segment on the side. So we have the paper summary. It's already saved in JSON. And we have the supporting research already saved as well. And we should start seeing some other, like the script. When the script is done, you should see the script being populated in here. There you go. So that's not the one. So this is the one. So there you go. It's saved the supporting research. And I mean, if you click on it, you should be able to see. So human feedback, it's asking for human feedback here. It's asking me, What do you think about this? Does this look good? So let me go through it real quick. I think it's good. So I'm just going to tell it looks good. Looks good. So it should continue after that since I've told it it looks good because it says human feedback, provide feedback on the final result and agent actions. Respond looks good to accept provided request. You can. Yeah. So I told it looks good. So you went ahead and passed it to the next agent. So let's see, it ran into some errors. I probably changed some elements there. All right, so it's starting to write out the content, which is great. If we look at the content. Here, it's already starting to write those individual files that I mentioned. So it's sending the individual transcripts. So if you look here at enhanced podcast, you could see this here and let's format this. So you could see there's like this here you have Julia and Julia is talking and this is the text that Julia is going to say. And this text is what Guido is going to say. So it's going through all of this and picking each individual one. And sending it to ElevenLabs. ElevenLabs generates the audio for it and saves each segment of the audio here. Once it's done going through all this and generating audio for them, it's gonna put all of them together and generate one cohesive MP3. And we're gonna say it's gonna save it here under podcasts. And that should be the final one. And you can actually, we can actually listen to that. So even, even if you get the, like this first one right now and download it and listen to it.

Hey Guido. Did you ever imagine chatting with a database like you're catching up with an old friend? That's exactly what these function calling models let us do. It's almost like having a conversation where your words translate directly into SQL magic.

There you go. So you can see that's the first one, how the first one sounds. And here's the final podcast now. So let's see.

Hey Guido, did you ever imagine chatting with a database like you're catching up with an old friend? That's exactly what these function—

and that's like 3 minutes long—

function calling models let us do. It's almost like having a conversation where your words translate directly into SQL magic. Oh really?

So it's not just about text-to-SQL anymore, it's actually like talking it out and watching your query come to life?

Exactly. In the study Function Calling for Natural Language Database Access, they show that models like Claude 3.5 Sonnet and GPT-4o convert our natural language into perfectly structured queries. Imagine throwing filters, Boolean operations, aggregations, and group by into the mix and seeing it all work out seamlessly.

Wait, that's interesting. I saw that Claude 3.5 managed a 74.3% exact match and GPT-4.5 managed a 74.3% exact match.

All right.

That's all I wanted to share, how to build that real quick and how to make that work. So Connor, I'm going to be sending you this. I'll probably upload it somewhere and send it to you so you can listen to it. Hopefully it's I mean, you could be a good judge of how it did. Um, but that's just how quick I wanted to go over this. That ends my stream for now. I might do some tomorrow or later. Uh, but yeah, thank you all for watching, and I'll see y'all later.