Building LLM Applications w/Gradio

I prepared this Hugging Face space that shows all the different ways that you can build a chatbot with Gradio and for different cases of where your model is running, whether it's running on a separate API server, whether it's running locally. I can show you, this is a server.

So people can play around with it as well?

Yeah, yeah, I posted it in the— in my Hugging— in the— my Hugging— in the— uh, don't switch. I put it in the office hours. Yeah, in the Discord. Um, I put it here, right? Yeah, yeah. So if people want to follow along, they can— they can go here. Um, but yeah, so this is like a chatbot, you know, like, uh, you know, how do you do, uh, a for loop in Python? And it'll— should stream This is like a ChatGPT UI right here. There's functionalities for retrying that, undoing it, clearing the whole chat. This is a full UI that you could use to build an app around your LLM. This is the code that goes into it. As you can see, it's only you know, 50 lines of code. So it's not— and it's all Python. So yeah, it's pretty, pretty quick to get started. Yeah, so in this case, this is like interfacing with, you know, it's using the Hugging Face like inference API to query the LLM. But I'll show you guys an example with Transformers, like if it's running locally before you deploy it. But yeah, basically what this is doing, it's creating, you know, a Gradio chat interface. So this is like a, like a one-line UI for your LLM. And in order to implement one, all you need is this function here, uh, respond. So respond takes, uh, um, it takes one, uh, basically two parameters, um, at minimum. The first parameter is like the current message that was typed in the text box, and then the other message, or the other parameter, is like the list of all previous messages. And they're, you know, right now we don't support the OpenAI format. We support, you know, we have like a, they're passing us like a list of tuples, but we're working on supporting the OpenAI message format shortly in, you know, probably like about the next 2 weeks or so. But yeah, so you just, you know, create, you know, format the history and then you just feed it to your API and then you just yield each successive token. You know, you append it to the response and then that's how, you know, like the, it's just like a simple for loop and that's how you get like the streaming that you see here. Yeah. So like this, this tracks like all of like the message history and all that stuff. So yeah, so it's like, you know, once you have your model trained, if you wanted to share with someone, create a UI around it, it's— you can do it in about, you know, 50-ish lines of Python.

That's super cool. And you get to do this stuff for work.

Yeah, so a lot of what we're doing at Gradio now is making this even easier. How can we make the chatbot better? How can we add more components and stuff? Gradio isn't just for chats. There's everything, like image generation models, there's image components, video, audio, 3D modeling, all that stuff. We can build it in Gradio. It's all about how we can make Gradio the easiest library for creating any AI app purely in Python with as little code as possible, with as few additional concepts introduced and stuff. This whole thing is built in Gradio too, which is cool. This demo, this is a Gradio demo. There's a code component, there's all the stuff, tabs. You can create pretty complex stuff with just Gradio.

Awesome. And I actually shared earlier in the Discord a tutorial you did on a YouTube tutorial on multimodal chatbot stuff as well, which is—

Yes. Yeah, exactly. If there's, I can show you some multimodal stuff. If there's questions around that, I can show how you do that. Yeah, with Grigem.

It would be fun to get to that. We've got a bunch of questions and two of them, which people have upvoted a lot, was gonna be my pretty much my first question also. It's— it revolve— they revolve around the— people have mentioned Streamlit and Shiny in terms of comparisons. There's my— I mean, a more general question is around the competitive landscape because we do have Shiny, R, and Shiny can be used from Python. We've got Streamlit. We've got, I think, Dash, and people do things with Voilà from notebooks, and people can, like, build stuff from the ground up using Flask relatively straightforwardly these days. So I'm wondering perhaps you could tell us a bit about the landscape and why people would choose Gradio.

Yeah, certainly. I mean, I think like all tools are great. Like, I'm not here to say that, you know, whatever, like, I'm not here to bash any other library that makes it really easy to build a UI. I will talk about like some of the things that we think are, you know, one of the things that, or some of the things that make Gradio really nice. One is just that it's like built for like AI first. So there's a lot of really high-level abstractions that make it really easy to build the kind of things that as an ML researcher, AI, ML engineer, that you would want to do. So the first thing is this chat interface, right? So it's one-line full chat UI with all these different messaging abilities. There's the ability to— it's not shown here, but you could actually make it so that you can upvote and downvote messages, there's all these things that you would want to do for chats. Gradio makes it really easy to do that. I think that's one of the things that makes Gradio really great. It's designed for AI first. Some of the other things that I think Gradio is really good at is the API, the built-in API usage. Let's say this is an example where you have— this is all Transformers code. This is a the same chat, but it's built with Transformers, is running locally. Now you have a UI, but let's say that you also wanted to use this model as an API because you wanted to use it elsewhere. You want to just use it via API, you don't want to use it via the UI. Gradio comes with a built-in API. If you didn't see that, if you scroll all the way to the bottom, there's this use API. Then there's a Gradio client library that comes installed with Gradio. This shows how you could just query the chat with Gradio and then you get a full API. It's not just for these chat applications, for any Gradio application, it's like a built-in API that you can use from anywhere. Then some of the other cool stuff about Gradio is that the integration with Hugging Face is quite nice. If you have a Gradio demo that runs on Hugging Face and uses PyTorch, you can use this thing called ZeroGPU, which is like a shared sort of GPU cluster, and it's free if you're like a Hugging Face Pro subscriber. So this demo that's running on a, you know, it's running a Transformers model locally on this computer. You know, like, I'm not paying for the GPU right now. So if you have like, you want to deploy your model that needs a GPU, you can use Gradio and Hugging Face to do it without paying for the GPU. So that's like another, like another cool thing about, about Gradio. But yeah, and I think, you know, we're, we're making Gradio better every day, but those are like some of like the, the, the 3 things that I think, you know, make it, make it useful, particularly for like this, like AI, like use case.

Absolutely. I'm glad you highlighted the integrations with, with Hugging Face, because that's one of the big value props for me. So many, I mean, I just, Hugging Face is such a wild space for me, in, in all honesty, like the models that and datasets that come out all, all the time. It's so much fun to explore. And the fact that it integrates so nicely with Gradio is a huge part of the value prop. So that's super cool. There is a related question. Someone's interested in learning Gradio and migrating from Streamlit. So I don't know whether you have any, any thoughts upon what that process is like or any advice you'd give there.

Yeah, we don't have a dedicated migration guide on the docs or anything. I think it's very similar to Streamlit in the sense that the UI is declared. In this case, it might be a little bit clearer. This is using a different sort of radio API called the Blocks API. And you can see that the UI is it's declared like the UI is a declarative UI API. The UI is rendered like how you see it here. First, it's like a markdown that shows the title, then the chatbot, then the text box. It's very similar to Streamlit. It's very declarative. Though the one difference is that in the way that Gradio knows when to run your Python function, you have to tell it when one thing changes, run this function. So that's what's happening here. Like when someone submits a prompt, run the respond function with these inputs. That's like the one difference with Streamlit. You have to be a little bit more like imperative, I guess, with the, you know, the reactivity. But the benefit of that is that you get the— that's how we know how to build the API for you. And the other sort of benefit of that is like Radio only updates the things that you tell it to. So that it actually helps with performance as well. And yeah, like Gradio, you know, if you ever go on Reddit or whatever hacking news, you hear people talk about Gradio, they'll say like, oh, it's only for like, you know, these sort of like demo toy use cases. That's something we're trying to like push against a little bit because you can serve like a lot of traffic from a Gradio UI. You know, like the LM says leaderboard, for example, built entirely on Gradio. They handle something like it's something crazy. I think it's like several thousand, tens of thousands of requests a day. With a Gradio server. So, yeah, like, Gradio is actually pretty, pretty performant. So, yeah, so that, that's sort of like the main difference. You have to be a little bit more imperative with the reactivity. But apart from that, they're, they're similar in terms of like the UI, like, design philosophy, I would say.

Yeah, cool. The example you showed had streaming components, and Ruben has a great question. Does it work with streaming out of the box?

Yes. Basically, you can stream anything, not just text. The way you do that is just from within the function that you tell Gradio to run, you just implement a generator. Then Gradio will just know to just stream that function, like the contents to the UI. It's not just text, you could do it for a diffusion model. That's something we've seen a lot. People like stream the results of these fusion models. You can see like the image go from like white noise to like the actual image at the end. It's like a pretty slick UI. Yes, you can do that with, yeah, with like images in that case. You could also stream from your webcam and stuff like that. So if you're doing like an audio streaming demo like with Whisper or something, like you could do that as well. You could like stream to the server and back.

That's super cool. Kit has a great question. Firstly, Kit says, first, bravo, and thanks for the great work. You really nailed the core use cases. Could you take us— this is nice— could you take us briefly through the Gradle repository, outline the moving parts of the code, telling us roughly how it works?

Yeah. Yeah.

Perhaps tell us where contributions make sense to you, what's needed, where it's going. So yeah, that, that type of overview could be, could be super cool.

Oh. Yeah, for sure. Yeah, so Gradio, it's a little bit different from other Python open source packages just because there's a big JavaScript component. But yeah, basically the way that the library is laid out, it's like two main directories that as a newcomer you would want to familiarize yourself with. The first is this Gradio directory, which that's where all the Python code lives. And that's all the source code for the components. So the RadioTextBox, RadioImage component, all that stuff, as well as the FastAPI server, all that stuff is in here. I can go into that in a second. And then also the JavaScript is the other main half of it. And then the third component is the client that I can talk to about later. I can show we're going to 1.0 the clients in about 2 weeks. So we can talk about that later if people have questions about the clients. But yeah, so this is like, you know, like the Gradio sort of source code, right? So, you know, there's like a CLI for developing like custom components and stuff. That's like the other cool thing with Gradio. You can like create completely new components that run, that work the same as like any core component. So that's what this CLI is doing. But most of the logic goes here. And so like these are all the Gradio components, like the checkbox, the code, all that stuff. Then I could show maybe the image, for example, the image, probably one of the most popular components. This is how a component looks like. These are all the events that are defined for a component. You can stream an image, you can upload an image. When you change an image, that can trigger an event. and, you know, these are like the constructors and stuff. And, um, like the two main, um, the two main, uh, methods that all components implement are like this preprocess function, uh, that basically that takes the data from the server and that kind of puts it into like a Python-friendly format. So in the case of image, you know, you can, you can select to receive an image from the server, or like from the front end, I mean, as like a NumPy array, a PIL image, or a file path. right? So this is kind of what handles that logic. And then the post_process does the opposite, right? So like, you can, if you want to update an image, you can return like a NumPy array, a PIL image, or a file path, and then Gradio will turn that into something that the UI can display. So those are like the two main methods that every component has. And then like the constructor obviously has like all the, you know, all the all the constructors and stuff. So that's sort of like how all the components are defined on the backend. And then, you know, the other half of the coin is like in the JavaScript, right? So Gradio is built with Svelte on the frontend, which is actually pretty easy to learn. I didn't have a lot of experience with it when I started working on Gradio, but I feel pretty, you know, after, you know, after a couple days, you kind of get up to speed. But yeah, so this is how the checkbox is implemented in Gradio, right? So js-checkbox, and then this is when all the events are dispatched. So whenever you see gradio.dispatch_change, that's when the change event of the checkbox is triggered, and that's when all their Python function will run, basically. So that's kind of a high level of how everything works. I can go into a little bit more detail, but I'd rather cover like a little bit more high-level stuff. You know, we don't go into like all the weeds here because, you know, to build a demo you don't have to know all this stuff. Um, but, um, but yeah, and then like there was one more thing about like contributing. Basically, like, um, you could contribute to, you know, any, any sort of issue would be welcome. Um, you know, there's like a lot of like, yeah, pretty much like any issue, uh, would be good. Like some are sort of backend issues and we try to label those as like being a backend issue. This, this could be like a good example here, right? So this adding like a delete event to the file component. So this would cover both the backend and the frontend. But yeah, there's, there's like lots of issues that, you know, it's, we're having a hard time doing everything that we wanna do. So if you wanna help out, just comment on the issue like, hey, I'd like to work on this and then it's, it's yours.

Very cool. And it's great to see the good first issue label as well. Super, super useful. That, that was awesome, Freddie. We've, and if, if later on near the end we wanna get deeper in, in the weeds with this type of stuff, we, we can, but I, I do like keeping it, keeping it at a high level. We've got, we do have a question around multimodality. So anonymous attendee asks, what are some of the cool features that Gradio supports for audio and multimodal? Is there only turn-based or is there streaming there too?

Yeah. Yeah.

And then will be needed for future GPT-4.0 model like UX.

Yes. So yeah, the GPT-4.0 thing is something that we're working on now, how to like make that really slick. But yeah, like for like multimodal, like chatbot kind of thing, like Radio has a couple of good components for that. So the first thing I'll do is show you the multimodal text box. Gradio has this multimodal text box here. You could use it to send both text and attachments to any demo, particularly for— but it works good for chatbots and stuff. You can attach images or any file and then you can say, what's in this image or something? That's just like a a multimodal text box. So that's what this component is. But obviously it's really cool when you combine that with a chatbot. So I'll show you what that looks like now. So multimodal, I think it's multimodal chatbot, uh, yes. So, uh, this is like a, you know, it's not interfacing with an LLM, so the response will be like really really silly, but what do you think of this? Yeah, that's cool. Then this is what I was talking about earlier. You can define how to go down. Thank you. But yeah, so this is how you can build a multimodal chat with Gradio. There's this multimodal chat box, text box component, and then you can just— I'll show you the code now. It's pretty similar to the other ones. You define a text box and then you can say— sorry, let me move this out of the way. You can say what you want to accept images or what. Then when you submit something to this text box, run this function and then that'll run this function here. That'll add it to the history and it'll also clear the multimodal text box. and then it'll— in this case, it always responds with, "That's cool." So that's why I asked it what I thought, because I knew what the answer would be. Um, but yeah, so this is, uh, I'll ping this, I'll put this in the chat. This is our— in the Discord. So, uh, multimodal, multimodal chatbots. Um, and then also, if you want to write even less code, um, you can make the chat interface multimodal as well with just one parameter. I think, oh no, it's broken. I don't know what's going on there, but basically I'll get this up and running. But basically if you want to have a multimodal chat interface, all you have to do is set multimodal true and that'll create the same chat UI as you have here. But it'll let you upload like files in this text box as well. And then it'll, you know, display everything and all that stuff. Yeah. So I'll put this in the Discord as well and then I will, I will unbreak it.

Okay. Sweet. Thanks, Freddie. Got a great question from Andrew Grandgard. Can, should I ship Gradio-based API or maybe apps to production?

Yes. I mean, I think it depends on exactly how heavy it is that the processing that the Gradio server is going to be doing. But yeah, I would say you can ship very performant Gradio demos that handle a lot of traffic. And yeah, we also have a guide around how to do that. So I will share that here.

That's cool, because there's another question, which is, is Gradio meant for commercial use? Any tips on deployment and advanced usage tips for Gradio for production purposes?

Yes, I will. Yes. Where is it? Here we go. Set up a demo for maximum performance. Yes. So let me put this. Tips for Gradio's max performance. Sorry, let me— OK, I'll answer the question a little bit more. But yeah, I think Gradio can definitely handle a lot of— it can definitely handle a production use case. If you're running a very heavy GPU-intensive task, Gradio has a built-in queuing mechanism to try to handle that load for you. You can make it such that, or you can control the amount of concurrency for that task so that you don't run out of GPU memory, for example, or that the server doesn't crash or slow down or anything like that. There's a lot of tricks around that, around how you can do that. Also, the other pattern that we see work really well is you can spin up a lot of these independent Gradio servers. and then just like load balance between them, and that works pretty well for a lot of, a lot of use cases as well. Um, but yeah, I think like, you know, like DALL-E was created with like a Gradio UI. Um, like I said, the LMTest leaderboard, um, runs like with Gradio, so, um, you can definitely handle like a lot of traffic.

Awesome. Um, so we have, um, a few more specific questions. So Shridhar asks, um, in Gradio window, the LLM chat, gradio.chatbot, is it possible to display some buttons? For example, predefined topics or questions to initiate the chat with the LLM?

Yes. So not like, yes, yes. Gradio has this thing called examples. So I'll show you guys a, like, like a Mistral, this one. You can see here, here's some examples. These are some sample prompts that you can use to pre-fill. I think this is what you're getting at, how can you give my users an idea of how to use this? Gradio has these constant examples which are basically just what are some example input components that I can let my users try in my UI. It doesn't just work for chat. Any Gradio demo has this concept of examples. There's an examples component that you can use to seed the demo with data. In the case of multimodal, you could provide sample prompts and images and stuff like that. Then Gradio can also cache the outputs of these examples. If you click them, the answer will also show up as well. So that can help in the case of people— you want people to know how your model behaves without actually using any of your GPU resources, for example, without actually running anything. You can tell Grader to cache the results of these. So I can show you what that looks like in the code. So in this case, it's just an extra parameter here to the chat interface. It's just examples here and then you just provide your examples. In this case, we're not caching the examples, but you can just set this to true and that'll do the caching mechanism that I was talking about earlier. I think that's what the— hopefully that answers the question. Let me know in the chat if you had something else in mind. But you can definitely provide some sample inputs for your users so that they have an idea of how to use your Gradio app.

Very cool. We've got a nice question from Lauriane as well. Lauriane's more comfortable for UI with JavaScript. So what are the current disadvantages of relying on the Gradio JS client and custom, for example, Svelte frontend components?

Yeah. So the JavaScript client is, you know, with like, you know, two of the engineers on the team, Pete and Hannah, have been working pretty diligently and hard to get the JavaScript client ready for 1.0 in about 2 weeks, or like, yeah, I think like 10 days. So June 6th is the 1.0 release of the client. So by then, like, the JavaScript client will be pretty, pretty, pretty solid. Um, and we've already seen like a lot of like really cool community demos being built with the, with the JavaScript client of like, you know, like you don't want to use the Gradio UI, you want to use your own UI, but you want to like use a model that's hosted on a Hugging Face space. You can like basically put that model in your UI with the JavaScript client, like that it can bridge the gap between your UI and the Hugging Face Hub. So we've seen a lot of cool demos built with that. So yeah, like in the past, JavaScript client was a little bit like rough around the edges, but we've completely revamped it. It's a lot easier to use. So I'll say definitely try it out and, you know, let us know if you run into any issues, but, you know, from what I played around with it, it's pretty slick. And then like the other, the second part about the question, like, you know, like what if you wanna use like your own Svelte code? Certainly, you can definitely do that, but I think the cool thing about Gradio is that we did all the hard work around defining a component library so you don't have to. Even if you're a really good front-end wizard, I always find it really annoying to create a good checkbox from scratch or messing around with the flexbox and making sure the UI looks actually really good and things are in proper rows and columns and all that stuff and making sure that the height properly fills in the entire content of the page. That stuff is annoying for me. Gradle can handle that for you. I think Gradle is a really good place to start. Then if you always wanted to do something a little bit more custom, you can create your own Gradle component. That's something that we shipped a couple of months ago. But let me go back to the app. We have this custom component gallery. These are Let's say you want to do something that we don't have support for in Gradio Core yet. You can create one of these components. This one's actually really cool. It's this rerun.io. Someone created a rerun.io viewer, which is this multimodal data viewer entirely in Gradio. That's really cool. Gradio can do a lot of the boring stuff, and then for the fun stuff, you could just write some minimal Svelte to do the cool stuff you want to do. And just plug it into the rest of the Gradio demo. People can use this from Python, which is pretty incredible. That's something that if you're comfortable with frontend, Gradio might still be useful for you in that regard. I think you got muted, Hugo. Sorry.

This is a really cool gallery, man. I was just wondering if maybe you can show us a few more of them by any chance.

Oh, yeah, for sure. So some of the cool ones. So like the PDF one. So I really like it because I made it. But, but there's like this PDF component. So, you know, like, let's say you have like a question answering like LLM, right? That like, you know, like document QA, like you could use this PDF component to like, you know, like answer questions about your, you know, your PDF, right? So supports multiple pages and stuff like that, and like it'll render the PDF completely like in the browser for you. And then some of the cool ones are—

Just quickly for the PDF one, I'm able to plug that into any model?

Yeah, I mean, like, like if your model accepts a PDF, like you can plug that into it. Yeah, yeah, yeah, yeah, yeah, for sure. Like you'll get a PDF like in the Python and then just feed that to your model. Yeah, these things are like, I think it's really cool. You just pip install Gradio PDF and then you have a PDF component that you can just place in your Gradio UI. Very cool. Yeah. Then some of the other cool stuff. This is really cool. I don't actually even know how to do this properly, but someone made like a molecule viewer in, in Gradio. So yeah, like I don't even know what the right input is for this, but I've seen tweets of people using it. It's really cool because it'll visualize like a molecule like completely in the Gradio and like you can like tweak it and like, you know, change it, edit it, that kind of stuff. So, you know, if you're really into like biochemistry or that kind of stuff, like you could use like this component. Like this model or this is really cool because it's basically like that search box you see in the Hugging Face Hub. This is basically that as a Gradio component. Let's say you want to be able to download a list of models or something from the hub in your space. You could use this component to do that and it has all the it has the really slick autocomplete, so it's really cool. Yeah, so there's a lot of stuff. This one's good for geospatial data.

I'm a huge fan of Folium. I've been a fan of Folium for about a decade now, actually.

Really? Yeah, it's really cool. Yeah, so you could use this to embed any kind of map into Gradio. Yeah, so there's a lot of cool stuff here. That I think certainly, like, people aren't, you know, people aren't using yet. Yeah.

So that's beautiful, beautiful stuff, Freddie. I'm really excited to go and explore this stuff a lot more myself. Someone in the Discord has written so many ideas to explore. I think the course will leave me with 500 days of homework, which I like the 500 days. I would encourage people to not consider this homework. This doesn't even feel like work, right? This is so much So much fun. Anand has a great question. Would you recommend Gradio for building multiple users' GenAI apps on the cloud? And also in general, should GenAI apps be running on a serverless architecture?

I don't know what you mean by multiple users in that question.

Oh, yeah.

Like, yeah, certainly multiple people can use, like, a Gradio demo, like, concurrently.

I think that's what it, like, means, like, how many, I suppose, how many users would it support? Or how, like, how to scale it?

Yeah, I mean, it depends a lot on, yeah, like, what exactly, what hardware you're using to run the Gradio server. Like I said, like, you can tweak Gradio to be, like, as, you know, you can tweak Gradio to the specs of your machine, right? Like, I was talking about this thing called, like, the queue earlier. So, like, Gradio by default, like, kind of queues all the incoming requests. and then by default it'll only run like one GPU-heavy thing at a time, right? So that's done because by default you don't want to run on GPU memory. But let's say you have like a really nice GPU, like you could crank that up such that you run multiple of these things at a time. You can also batch requests with Gradio. So you could like, you could like handle multiple like incoming requests at a time. So that can help you build a more performant demo. But yeah, like, it depends a lot on, like, exactly, like, what hardware and stuff that you have going on, you have at your disposal. Certainly, like, you know, if you want to, like, host your LLM in, like, Modal, like, on, like, Hugging Face, or, like, you know, Replicate, something like that, and then just, like, query from Gradio via, like, the API, like, certainly that would work too. And in that case, like, the Gradio server would be, like, a little bit more stateless. In that, in that regard. But, but yeah, like, I think both, both patterns would work. It's just a matter of like what your, like, what your budget is, what you're comfortable with, all that stuff.

Yeah, it makes sense. And so we're, of course, we have our Discord, but like downstream future music, if people wanna like learn more about this type of stuff or chat with the community, is, is there a good place for people to do that?

Yeah, for sure. So the, the Hugging Face, Discord has a pretty big Gradio community. So there's a Gradio— this is a Hugging Face Discord. I'll post a link in our Discord later, like the join link. But yeah, there's this Gradio questions channel. A lot of community members just answer each other's questions here. It's actually really cool. And we make a lot of Gradio announcements here. And we have this Gradioverse community channel for people to just share you know, what they're building, what they're, you know, what they're having trouble with and stuff like that. So I, I would say this is the, the best place for like, like the Gradio community. And certainly if you have like a feature request or a bug or something like that, like just file that on the, on the, on the repo and then someone will, will get to it for sure. Awesome.

Thanks, Freddie. Got a question from Colin. Have you seen any interesting designs for multi-agent collaboration visualizations? And Colin says, Colin's been messing around with agentic workflows and be cool to mock up dashboards quickly when tweaking interactions.

Yes. So that's like a really cool topical question. That's something that we're working on now is implementing like the, the agents, agent workflow with Gradio. And like, I will show you guys actually, actually built a custom component to do that to sort of test out like the API and all that stuff. So I'll show you that now. Yeah, so this is like an LLM agent running on Transformers. So Transformers released an agent API, so you can kind of build an LLM agent with the Transformers library. And this is sort of like a UI for that. So like, you know, like, show me a picture. Let's see if I could type. Qcat. So I hope this, um, so we should see it soon, hopefully. Um, yes, so, um, so right, so like here you can kind of see like the, the chain of thought of the, of the agent, right? So it's like, okay, I need to generate a Qcat, I'll use this image generator tool, and then it shows you the tool that it used with like the prompt, and then it shows you like what the image was, right? So there's like a, like a fully functioning like LLM, you know, if we go to the space, I can show you the code. So like I said, this is a custom component, so right now it's not part of Gradio Core, but if you install this, you can use it for the time being. So yeah, so there's like an agent chatbot component, and then this is the code. So it's a little bit more involved because, you know, there's like a bunch of tabs and stuff, but at the core, it's basically what I showed earlier where there's like a chatbot, a text box, when you submit, run this Python function. So that's how you can do the, the, the sort of the, um, the, the agent stuff. It works with LangChain as well, so it's not just tied to Transformers. Um, but yeah, so yeah, definitely like agents is something we're working on right now. It's like you can consider this like a pre-release in this custom component. We're working on adding that to Gradio Core shortly. Um, multi-agent Certainly you could do that, but it's not like— right now the Grado chatbot assumes there's only two speakers, but you could think of ways of maybe depending on speaker, in your message you'd say this is whichever speaker or something like that. But certainly a multi-agent chatbot could be a really cool custom component that someone should build. If someone wants to do it, they should do it. We would love to communicate that and share that. But yeah, so like the multi-agent, multi-speaker is not something that we've built yet, but the agents we're working on, that should land in core shortly.

That sounds awesome. And so Colin, that sounds like if you have the time, creating a custom component could be super cool and let us know how you go. Would you mind putting that in the Discord as well, Freddie?

Yeah. Yeah, for sure. For sure.

And it was a really cute cat. As well.

So, um, visualizing. Yes.

So Matthew Miller has, um, a bunch of cool questions. Uh, is there a plan to enable authentication in Hugging Face Spaces that doesn't require allowing cross-site cookies? I haven't experienced this issue myself, but can't remember exactly what the issue is. It currently doesn't work for Safari mobile, for example.

Yes. Yeah. So in terms of authentication, there's like two sort of things that you could do that are a little bit more robust to the built-in Gradio authentication. So yeah, like the, the Gradio authentication is kinda like, you know, that's sort of like a, it's not really, we're, we're, we're thinking of ways to, things to call it that's not authentication because it kinda gives people the wrong idea. It's not super secure, to be honest. It's just kinda like the minimal, like, access control you can do. But one thing that's really cool is that there's like a sign in with Hugging Face button that you can use to put it in your, in your Gradio demo. So I'll show you a, um, um, I'll show you a, so a GGU MySpace, I think. Uh, sorry, let me think of a cool demo. So I don't know if you've seen this, but, um, This is a demo that once you have your fine-tuned LLM, this will turn it into GGUF format so that people can run it. It's the quantized format. This will do it all with this Gradio demo. You can only do that with your own models, so you have to sign in to Hugging Face to do that. You can do that with Gradio with this local button here. It tells you what permissions it's going to ask for. You can say whether or not you want to do that or not. Yes. So grant, and then I'll do it in my Gradio space. Cool. So yeah, so now I'm logged in to my Hugging Face account. So if you go to the code, it's just like another Gradio component.

For those who don't have a lot of context around GGUF as well, GGUF is what has helped me run models locally on CPUs also. And it's been a game changer, in all honesty.

Yeah, yeah. So like, you know, like let's say you've, you know, at the end of the course you have a fine-tuned LLM, you want to run it on like a CPU, you want to convert it, you know, you can convert it to UEF, you can do that all with Gradio on the hub. So this is, you know, I'll share this as well on the Discord. But yeah, I wanted to show you guys because of like this like login functionality here that just, it's, you can use that here and it'll use OAuth. So it won't use cookies or anything or like, yeah, like the cross-site cookies. Yeah, and it's just like this, like, you know, let me go all the way down, login button right here. So it's just like a very simple radio component. And then there's also additionally you can add like Google OAuth to your demo as well. So let me see if I could find that quickly. But yeah, so there's more than you don't have to just use the cross-site cookies like we've, we've we've, uh, yeah, we've, uh, like, sorry, I lost my train of thought, but yeah, we, we built like other sort of like authentication mechanisms into, uh, into Gradio. So I think, uh, let me go find that OAuth here. I think it's, um, I'll find it later. I can't, I can't find it right now, but yeah, like there's a, there's an example here of using Gradio OAuth in your, or Google OAuth in your Gradio demo. So I'll, uh, I'll show that. I'll put that in the Discord at the end. But yeah, it's certainly possible.

Cool. What a question around does Gradio integrate with FastAPI? I'd love to know that. But I think more generally, just how do you all at Gradio think about, you know, the serving story?

Yeah. So Gradio is built upon FastAPI. Like, Gradio is a FastAPI server. It just shares— it just, like, serves, like, a very specific in the index.html file that has all the Gradio frontend stuff in it. The way that you can think of Gradio, it'll take all of this. Basically, Gradio will take all your source code and turn that into a JSON config and then serve that via FastAPI to your browser, and then the browser will run all the reactivity and stuff. Whenever something happens that requires Python to run, it'll send a request to the FastAPI server. FastAPI will run it and then send it back. That's like Gradio at a very high level, how the architecture works. Gradio is basically just like a FastAPI server. All that stuff when I shared earlier of the API stuff, that's just like the FastAPI server that's running it. There's nothing fancy, nothing Nothing crazy like that. One of the cool things about Gradio being built upon FastAPI is it's very easy to integrate it within a larger FastAPI application. FastAPI has this idea of subapplications. You could basically mount a full Gradio UI in a URL of your larger FastAPI app. If you have a larger app that does a bunch of other stuff beyond Gradio, you can still use Gradio to just mount it within your application. That way you don't have to deploy a whole separate server or anything like that for Port Gradio, I could just integrate very seamlessly into FastAPI. I could show an example of how to do that as well. I think in the docs down here, here you go, mount Gradio app. Here we go. This is how you can take, you have a FastAPI app and then you want to mount a Gradio at the Gradio path. This is how you This is how you would do it. Yeah, so yeah, it's pretty cool. Yeah, Gradio can— it's just basically like everything's a wrapper now. Gradio's a wrapper on FastAPI, basically, very, very cheeky way to put it.

Awesome. DMG has a question. How can authentication and authorization be integrated into Gradio to manage user access?

Right. Basically, um, Gradio— so when you, like, um, Gradio has this thing called, like, the request. So, um, you can add a parameter called request, um, to your function, your Python function, and then you will get, like, a FastAPI, like, request that you can, like, see, like, you know, like, who's logged in, all that kind of stuff. And then depending on that value, you can, like, say, like, you know, like, if this user is not who I think it is, just don't run the function, raise an error, that kind of thing, right? So you can, you can use like that sort of pattern to like control like which parts of your app are accessible to which user and that kind of stuff. I, I think that's kind of what the question was getting at. But yeah.

Yeah. Matthew Miller has another good question and it's actually around Gradio Lite and we haven't really talked about Gradio Lite.

Yeah.

So maybe you can give us a whirlwind tour of—

Yes. So Gradio Lite is a, you know, it's just a version of Gradio, but it'll run entirely in your browser. It'll, it's used Pyodide. So it's like serverless Gradio basically. So that's really cool because, you know, like let's say that you have like, you know, you wanna like transcribe audio and like you're worried about like privacy, all that kind of thing. You could do that basically entirely in the browser using like, you know, like Whisper Lite and, you know, and Gradio Lite. I can show— oh, sorry, here in the docs. You can go here to Gradio Lite. Then this is how you basically just add this to your HTML and then you just write your Gradio application within the HTML. But you're using Python, you're not using— it's just Python. Then Powdad will run it for you. Then that's how it works. Then you can separate into multiple files if you want and stuff, but it'll run entirely in your browser. Then there's actually some really cool demos of using Gradio Lite with Transformers.js. If you're not aware or if you're not familiar, Transformers.js is the Python Transformers library but running entirely in your browser and it supports a staggering amount of like tasks and models. It's pretty incredible, actually. And then, so like if you have like a Transformers.js model that you want to create a UI around, but you don't, you know, you don't want to build like a whole UI, you don't want to worry about like what component library to use, anything like that, you could just use Gradio Lite to basically run your Transformers.js model like in a nice UI in the browser. It's pretty slick. I'll try to find an example of that and put it in the Discord later.

That's incredibly cool. When we're thinking, like, Grady, like, when we're thinking about, you know, Python in the browser using Pyodide in the backend, PyScript comes to mind as well. So when, when would you suggest people play with one or the other?

Or I'm gonna be honest with you, I'm not the biggest expert in, like, what the state of Python in the browser is. But yeah, like I— yeah, so I don't know. I'll get back to you on that.

Yeah, cool. Appreciate it. The initial question Matthew Miller had though was, in Gradio Lite, is it possible to use Python packages that need to make web or REST API requests? If so, are there any examples?

There's no examples off the top of my head. It is possible. There's a very specific way to do it. You can't use requests, or I think you can use requests, But also there's like, like in the, in the Pyodide docs, there's like a specific section, if I recall correctly, about how to make requests correctly from within Pyodide. So I will dig that up and then put it in Discord. Yeah, but it, but it is, it is possible.

Appreciate that, Freddie. And no promises, but I, my spidey sense says if you join the Hugging Face Discord and did a search for this type of question, someone else may have asked it. Before. Yeah. Seems like the type of thing people would be very interested in. Matthew Miller also has another question, which I'm— is there a way to do more advanced table like AG Grid? I don't know what that means. Or similar that has more advanced native column filtering input output. I find the existing DataFrame component a bit limiting. Would be very helpful for building more advanced web apps that could be used for things like data labeling. For example?

Yeah, so, um, yeah, I don't know what AGTable is. Um, I did, um, like, we are sort of like, um, you know, like, we are trying to make the DataFrame component better. Like, it's pretty, it's pretty flexible. Like, pretty much like anything, it'll, it'll basically like visualize any kind of like data for a Pandas DataFrame for you. So like, even if you've like applied colors and formatting and stuff in Python, like the JavaScript will know how to display that correctly, so it'll apply the same colors and all that kind of stuff. But yeah, like it doesn't do like a lot of like high-tech filtering or like it'll sort, but it won't filter for you. Like it won't, it won't do that stuff. That would be like a great custom component if you feel motivated to do that. But, you know, if you, if you don't wanna start from scratch, There is something I want to show you that's related to that, which is the leaderboard component. This is something that we did. This is something that we built a couple of weeks ago because the OpenLLM leaderboard was running all of the processing, all of the logic for filtering and subsetting and all that stuff in the backend. It was introducing a lot of latency. We built this custom leaderboard component. To be able to do all the data processing entirely in the client. It goes a lot faster. It'll filter very quickly. It'll add columns very quickly, all that stuff. The OpenLLM leaderboard is now using this custom component because it's a lot more performant. It's doing all the logic on the client side now. If you wanted to, I think it'd be really cool if you could build whatever it is that you have in mind as a custom component. And you could also file an issue so that someone else could, you know, propose ideas of how to best do that with Radio. But yeah, but like, we are starting to think about that. We are trying to go in that direction. We just haven't gotten there yet.

Awesome. Thanks, Freddy. We've got a few minutes left, and we haven't got to all the questions, but people feel free for the ones we haven't got to. If you wanna ask them in Discord, we can get to them there. I'm just interested in what you're most excited about working on at the moment. If there's anything you want to show us or tell us, what's the next 3 months or 6 months or, you know, what the future holds for you?

For sure. For sure. I mean, I think, you know, like, one of the things that I'm personally really excited about is, like, the agent stuff that, you know, someone asked a question about earlier. Like, I'm really glad that they asked that question because, like, there's something that we were working on, like, literally last week, right? So I think that's definitely going to hit Gradio soon, probably in the next 2 weeks or so. Basically, be ready to build really slick agent demos, better integration into all these API providers, all that stuff from within Gradio. We're actively thinking about how to make LLMs even easier with Gradio. I think that's one really cool thing. Then earlier someone asked about a GPT-4.0 type demo where you're constantly talking with it and it streams all the data all the results back to you. We are thinking about how to do that with Gradio. A couple of months ago, we were playing around with WebRTC to see how we could basically do all the Gradio streaming for audio or video with WebRTC. Some of the preliminary demos that we built are really cool. They're basically real-time machine learning, image processing, audio processing. All within with Python. That's really cool. Jury's on whether or not we'll use WebRTC or we'll use some other technology, but I think being able to stream really quickly from your client to the Gradio server and back, I think it's something really exciting that we're working on. We definitely have that GPT-4o type demo as a North Star that we're working on. Then the other cool thing that's going to hit Gradio really soon is more declarative UI. I think this is actually really cool. It's something that— oh, sorry, I didn't mean to do that. It's something that we're working on pretty right now and we're going to release it probably next week or next week at the latest. But we keep getting questions around how can you create Gradio demos? That are in a way non-deterministic, you want to have a variable number of text boxes, you want to have a variable number of events and all that stuff. We're introducing this thing called gr.render so that depending on this arbitrary count variable, you can create that many number of text boxes that function similarly to any other Gradio text box. You can create click events and all, or like submit events for all these checkboxes. And then you have like a really cool declarative UI in Gradle. So I think it's really cool. I'll try to find like an example of that. Like this is like, it's gonna hit Gradle like probably next week. But this is like really cool, like sort of being able to do like really like dynamic rendering of like all these different components. I think that's really exciting. So I'd say those are like the 3 main things that at least I'm looking forward to in the next months. But the Gradio team is working a lot, right? So that doesn't cover the clients. The clients are super exciting too, right? So if you want to— the hottest models always land on Hugging Face first. And the way to use them programmatically is the Gradio client, basically, because you can interface with whatever Gradio server is running that model from anywhere. So I think that's really exciting. Gradio Lite is also super exciting. So yeah, there's a lot of exciting stuff.

Yeah, that's 3 incredibly exciting things, any one of which is kind of mind-blowing, to be honest. So what a time to be alive. We do have one other question. Someone's wondering— the first question is, can we have an LLM that is fine-tuned with the latest Gradio docs? I think that means maybe in the docs or because maybe an LLM that is maintained by HF. Every time they use ChatGPT, Anthropic, any LLM, they get Gradio code that's old and deprecated.

Yeah. No, that's, if you wanna build it, like, that would be so cool. Like, we are, we were actually talking about, like, doing that ourselves. Like, I, I mean, I'm actually enrolled in this course and the reason I enrolled in the course is because I wanna do that. So, but by all means, please do that before me. Like, that, and I would, I will help you. That would be really cool. Yeah, because we're seeing that, like, ChatGPT, all that stuff, like, they're using, like, really old Gradio APIs or they just hallucinate some stuff that like was never valid Gradio code. So yeah, so I would, I would love to have a Gradio LLM that's running on the website that can, or on Discord that people can ask questions to, you know, to unblock themselves. I think that'd be, that'd be huge.

That would be so cool. Look, I'd like to thank everyone for joining. We had, you know, around 100 people here the whole time with virtually no drop-off, which is, which is incredible. Which is testament to how exciting everything you talked about is, Freddie. But most importantly, thank you for sharing your wisdom and everything you're doing on the edge of all of this really, really exciting stuff. And if anyone has any other questions, Freddie, you can be around every now and then to chat in Discord.

Yeah, absolutely. I mean, I'm on Discord pretty much all day. Like, so, yeah. So feel free to drop questions. Yeah. We'll So someone will answer for sure. Yeah.

Fantastic. All right. Well, thanks once again, everyone, and see you in the next workshop. And thanks, Freddy, as well.

Yeah, absolutely. Thanks, everyone. Really appreciate the questions. Cool.

Thanks so much, man. That was really awesome.

Awesome. Yeah. Thanks so much for the questions. I mean, I was a little worried that no one would show up because it's Memorial Day, but I'm glad people showed up and had good questions.

There were people tuning in from India at 4:00 AM and stuff as well.

So that's, that's insane. Yeah, that's super exciting. Yeah. I mean, like testament to you guys as well for building such an engaging course that everyone is loving. I'm loving it.