FastAPI for Machine Learning: Live coding an ML web application.

Foreign.

Hi everyone and welcome to Fast API for Machine Learning, Live Coding, an ML web application. My name is Greg Lochmain and I'm head of product at 4th Brain. We appreciate you taking the time to join us for this event. We're so glad to see you tuning in from all over the world. If it's late for you, thank you for staying up to join us. Our event today will walk you through how to build your very own AI image generation web application using FastAPI and stable diffusion. We're going to get right into it today. If you hear anything during our event that prompts a question you'd like to ask Sebastian, please follow the slido link in the description box on the YouTube page. If you're just listening, check out the link to see questions others have asked and upvote your favorites. We will do our best to answer the most upvoted questions during the Q and A portion of this event. Without further ado, it's my pleasure to welcome Sebastian Ramirez to the stage. Sebastian, also known as Tiangolo, is The creator of FastAPI, Typer, SQL model and other open source tools mainly using Python. He's currently a senior staff software engineer at Forethought and also helps out other companies as an external consultant. Sebastian, off to you my friend. Take it away.

Thank you very much. Thank you very much for inviting me. Thank you everyone for being here. So amazing. Let's, let's, let's do it. So what we are going to do is that we are going to build well, we're going to live code. This live part is freaking me out. Let's hope it goes well. Please help me pray to the gods of live demos. We're going to live code a machine learning API with FAST API. So the idea with this FAST API is that we will get high performance. It's easy to learn, it's fast to code and it's ready for production. So first, who am I? I'm Sebastian Ramirez. I am from Colombia in South America. That's why the accent. But I live in Berlin, Germany and you can find me online on multiple places. I'm currently working at Forethought Plus I'm working as an external consultant with other companies and I created a bunch of tools as great Greg was saying. So we're going to talk about fast API. First, what is fast API? It's a Python web API framework. It currently has 50 something plus k GitHub stars. It's being used by a bunch of cool companies and the performance is in the top rank. Of what you can get in Python. So that's quite cool. And it's been like, it's been one of the most used Python frameworks by the official Python Developer survey, is the one with the fastest growth, is one of the most loved ones from the Stack Overflow Developer Survey. So it's been liked by a bunch of people, unfortunately. So that's why it will be kind of relevant. But then Fast API for machine learning people is what we are going to talk about. So I will assume that you know machine learning. I won't teach you machine learning. If you don't know machine learning, probably you go and take one of the courses at Deep learning AI or with a forest like those are the masters. That's where I, you know, like I started my career with the first course from Andrew Inc, on Coursera, before it was called Coursera, about machine learning, which is now what deep learning AI is doing. So you should probably go and study that over there. I will assume that you know the basics of web APIs like HTTP JSON, but we are going to cover all that here live. So this is what we're going to be talking about. Now, fastAPI is based on standards like open API, JSON, schema, OAuth2. The thing is that it's based on standards and it uses those standards underneath. But the way it's designed is that you don't have to know those standards to be able to take advantage of that. So you get things like automatic API docs like this. What we see on the side, we're going to see a little bit more about that. And it's all based on the standard Python type annotations. So from the type annotations you will get autocompletion, inline errors, a bunch of things. But then Fast API does automatic data validation data automatic data conversion, data validation and data documentation for you. So you get something that is based on standards for free, which is quite nice. Now let's start talking and showing slides like we came to live code and get nervous and type typos. So let's just do that. So I have here my editor and I am in this project that we are gonna work. So the first thing that you will have to do is to go and actually let me activate screencast so that you can see the things that I'm typing here. So the first thing that you will do in a project, it will probably be on the command line. You will create a virtual environment. So it will be something like Python MV ENV and then you will type the name of the virtual environment that you want to create. So you are calling Python with the flag M. This tells hey Python, I want you to use the module called V. V.M. is part of the Python standard library. And then we are telling it what is the directory that we want it to use to create a virtual environment. I already run this command, so I already have the visual environment here. Then the next thing that you have to do is to activate virtual environment. So you will do source env the name of the directory with your virtual environment bin Activate. This is on Linux or Mac if you are on Windows it will be slightly different. But the thing is that you can have to activate the virtual environment before installing anything inside of it. So I can activate that here. And then the next thing that you should do is make sure that you upgrade the version of pip. So you do Python M PIP install upgrade PIP and this will make sure that you have the latest version of PIP installed in your virtual environment. Many errors while installing packages and that you see like oh, error compilating, like compiling something or like building something. Those errors in many cases just because you have an old version of pip, but the error won't give it away. So just make sure that you have the latest version of pip. After that we can install the dependencies that we are going to be using. So we're going to use this file. Requirements. Where is it? I lost it. Here it is Requirements, requirements Txt I have a bunch of dependencies here. This is what we're going to use. There's a bunch of ideas that you can have about how to use dependencies, how to in them. We are not going to talk about that today. We're just going to install this like this. So we are saying we want to install fast API with the extra optional requirements. All this includes ubicorn, which is the server that is going to run fast API. We're going to use it in a second and some other additional dependencies. Then this is what we need for FAST API. And then we're installing the dependencies for the machine learning stuff. So about the machine learning stuff, this is what we are going to use. You are going to use the stable diffusions Stable diffusion model from Diffusers, which is part of Hugging Face. You can read a bunch about that in their blog post. I'm not going to be teaching you that we are not going to be reading all this. I already copied this code. So this code is not what we are writing because we are not writing the machine learning part. We are writing the API part. So we're going to be using this same thing. Then we have to install the dependencies for this as what we have here. So what we will do is PIP install error requirements and this is what we will type. I won't hit enter because I don't want it to reinstall the all the things I already installed all that. Then the next thing that we will do is to install the development dependencies which are this dev requirements, this file that I have over here and here I have just some, some, some packages that will help me lint the code and automatically format the code. And I also use this ipykernel package which is part of the jupyter Notebooks package is an internal part of that. And we are going to use it to create an interactive window while we are editing. So this is going to help us explore code a bit quicker. This is one of those secret tips that I think not many people haven't many use. And it's one of those amazing gems. So we have, after having all these requirements, now we can start coding. So let's, let's jump to the machine learning part of things. So I have this file ML. I just copied the code from that blog post and I just put it here. This is what we are going to use. So we're going to import some parts of the code, we're going to import some libraries and then for hugging face we need to have a token. So I put the token in this file token Txt I'm not going to show it to you because it's supposed to be private. But then I'm reading it in with this path and reading the content in this variable token. This is all that this is doing. Now let's actually explore all this thing live. So I'm going to select this first line and I'm going to say CTRL shift p to show this command interface and I'm going to say jupyter run. This is the option that I'm looking for. Run selection or line in interactive window. I actually have it with a shortcut of shift enter. So I'm going to hit this and this is actually going to start an interactive window here on the side. This is the equivalent of a jupyter notebook. So I have a kind of a jupyter notebook here that I can interact with and I can interact from the code directly. So let me hide this. Now we have this here and I can actually interact with this thing from here directly. I can create a new path. For example, just hit enter and Then it's going to create that object there in this interactive window. But I can select lines of code. Let me hide this on the side. I can select these lines of code and just import those different packages by selecting them and shift enter which is the shortcut that I have. And this is just going to send them to the notebook here on the side. Now this is important here. This is going to take a little bit because it's just importing the whole torch and everything. But after it's done you can see that we have the check mark there. Now we can select the token path and we are going to load the token here. Now we are also important here. We're importing the stable diffusion pipeline from Diffusers. This is part of Hugging Face. So we're going to create a pipe which is going to be a stable diffusion pipeline that is reading from a pre trained model. So I'm going to select this whole section of code and I'm going to run it. I'm going to send it to the jupyter notebook or the interactive window that I have here. The first time that you run this on your local machine this is going to take a while because this is actually downloading all the weights for this model and it's quite a big model so it's going to take a while. The first time you run it, I already run it. So this is actually being fast. It's just that the model is really big so it has to take some time to load everything. And you can also see that I don't have to select all the line. I can just like put the cursor on something. Oh by the way, you see these red squiggly lines? Those are telling me that there's probably an error in my code. Normally that's what those quickly red lines tell us. In this case it's just that there's a little thing that is not correctly written in the source code of this stable diffusion pipeline. Something that could be improved there. But anyway it still works. Also I don't have to select the whole line. I can just put the cursor anywhere on that line and as long as it's on the line itself I can shift enter. And this is going to send that to the interactive window. So you can see that this has been sent here. This is still going to take a while. So we're going to wait for a little second for that and we're going to start playing around with fastapi while this finishes loading our machine learning model. So I'm Going to create a FastAPI application here. We're going to do it from scratch. So we are going to create, we are going to import from fastapi, we're going to import the fastapi class and then we're going to create an app that's going to be a fast API instance and this object is just an instance of this app. Then we're going to create this app get and we are going to define a function that is going to be just this. Yes, thank you GitHub Copilot for the suggestions. So this is what, what we are doing. We are telling hey Fast API. Whenever someone sends a request that goes to the root path and it's an HTTP get request, an HTTP get operation or get method, I want this function to be the one that handles that. And I want you to return this content which is just this dictionary and this is going to be encoded automatically as JSON by FAST API. So we just write this. If you have never seen this symbol, this is equivalent of as if we save something like this, Hey, I want you app get, I want you to take this function read root and do whatever is it that you are doing there. What we are doing with that at single is roughly equivalent to that. So we are just saying, hey FAST API, I want you to do that thing that you do when you get requests to the root path. Do that with this function. Now we are going to continue with our code because the machine learning model is already finished. So we have this pipeline that was loaded and then we also sent the pipeline to Cuda. We want it to be executing on the GPU to be a little bit faster. So now we can play around with this. Let's actually move this a little bit to the side. So we're going to create a prompt which is this one. So I'm just. Let me move it to the top so you can see it better. And I'm just sending this to the interactive window so we have the prompt variable there. And now we can let me uncomment this line because this is the line that we are going to use. Let's select this line and we can execute it. And what we are doing here is, hey, I want to use this pipe to process the prompt. This is going to execute all these heavy machine learning models. That is going to take a while to process and to load and to do everything. And then this is going to generate some result. That result contains a dictionary that has a sample. And then we can take the first one of that and that is going to be an image. This image is some particular type of object. But this is what we're going is going to contain the actual image that we are going to be using. So in this variable images that we are going to have the generated stable diffusion machine learning image. Now the thing is, if we have this just here like this, it will mean that if in the main file we were importing all this code, then this will be running every time we try to import that code. So it will run every time we just load the module. And you can see that this takes a long while to process. We don't want this to be running every time, we just want it to run whenever we call it. So we need to create a function. That's what we are doing here. We're creating this function obtain image. And you can see that this is just doing exactly the same that we are doing here manually. It's just that we are doing it inside of a function and then we are passing the extra parameters that it can receive. So we are saying here's the pipe and this pipe is going to receive the prompt and it's going to receive some additional parameters. And these parameters, we are just taking them directly from the same function that we are creating to pass the parameters. The seed is just going to make. Actually this is done. Let's actually see what is the image that we got. So I can select this variable image and then I can send it directly to the interactive window and then you can execute that. And then we see, oh, this is a fancy astronaut riding a horse. Nice, very cool. And it's actually kind of flying. So it's. I guess the astronaut is actually flying the horse or something. I don't know. Anyway, now we can put this logic inside of this function. This is what we actually really want. So we can comment this part out. And then now the seed is just like what is the random generator number? We can fix that so that we can get always exactly the same image generated from some particular random number. The number of inference steps is how many times we want them all to process the thing to improve the results of the, of the generated image. If you lower this, we're gonna get a crappier astronaut, but it's gonna be a lot faster. And you can read about all these parameters on the, on the blog post. The thing is that we are just taking those parameters and just using them inside of the code here. Now we can select this function and we can execute it on the side so that we have the function available. And now we can just to check that this is actually working, we can uncomment this line and we can generate the image again. But using our new function that is going to be able to generate this image when we call it instead of just like whenever we import it. We are just saying, hey, I want the number of inference steps is just five. I want you to do some crappy random astronaut and I want the seed of the random number generators to be this one. So I want you to give me exactly the same astronaut, the same crappy astronaut every time. So we can just select this line and then we can execute it. And this is going to take a while, but it's not going to be as long as the first one because this is having a number of inference steps that is much lower. So now we can see what is the images that we got. And we can see some psychedelic weird random astronaut on some random weird horse. Nice. Amazing. This is all we need for the machine learning for the machine learning site. We don't want to generate this image every time we load this module. So we can comment this part out and we can just save this and we don't need this interactive window anymore. We already have the function that we need and we know that this is actually working because we were running each function thing step by step. We run this whole thing even once here. So we know that this is working. We can just like close this. And now let's actually play with Fast API so we have this code here. This is actually a completely valid fast API application. So we can now run this fast API application using UVicorn. The way we use UVicorn is that we call Ubicorn and then we pass this string main colon up. This string main colon up. That is the parameter that we pass to Yubicon is equivalent to this. Let's say that we are telling Ubicon, hey Ubicon, I want you to from main import the object app. So main is the thing that we are importing. Then after the colon is what is the thing that we are importing from that module. This app is the same app. If we go to the definition, it's the same app that we created here. So we are telling Ubicorn, hey, I want you to start the application using this object app. This is all this means. Now if we run this code, let me actually close this file. We don't need to save anything there. We run it and we say that Ubicorn is running and is running on this port, on this, on this IP and this port. So we can actually go and follow this link. This is going to open here and we can see that we are receiving. Let's move this to the side so we can see everything at the same time we are receiving the response that we were expecting. Hello world. Nice. Cool. That's. That's working. Okay, so let's actually try and do something else. We're going to get an item. Whenever someone sends a request to slash items slash and this thing is going to be a path parameter. So let's see what that means. But first let's create a function that is going to read this. I'm going to remove this because we just need the item id and we're going to return the item id direct. So whenever someone goes to let's say/example.com/example items/foo this foo, this will be variable. This will not be a fixed URL. This foo will go in this item ID that is put in these curly braces. The curly braces are the same ones that you use in F strings, but this is not an F string. This doesn't have the F here and it cannot have the F because this is not a variable here. But you can see that it uses the same curly braces. So you can remember what is the format that you need to put there fast. API is going to read this URL and it's going to extract the part that goes in this part of the path. It's going to extract it because it knows where to extract it from this definition here. And it's going to put the value of this foo inside of this parameter. We are telling it that this is an integer. So this is actually going to be like an integer like this. So then we are going to have, inside of this parameter, we're going to have an actual integer. So now let's actually save this. And if we go in the URL here, we go to items seven, it's going to say, hey, this is not found. This is because when we started Ubicorn it was actually with the old version of the code. We haven't restarted Ubicorn reading the new code. So we have to stop ubicorn with Control C or Control C also on Mac. And then we can start it again. But now I'm going to use this trick of dash, dash, reload. This is something that you should not use in production. This is only for development. But this means that we are telling Ubicorn, hey, I want you to check the files and whenever I change them, I want you to restart automatically the server. So now this is loading and now I can refresh this page and I can see that now I'm receiving item ID 7. And because this is reloading now I can change things here. Like let's say I want a message and the message is going to say hello world with a single code. So we have this message hello world. And if I save it, this is going to reload automatically. So let me save. And you can see that this, you can see the hazard warning watch file detected changes reloading. Blah blah blah. Shutting down. Starting. This is now restarted automatically. So now we can refresh this page and we can have the new message hello world that we were expecting. So nice. This is working quite nicely. This is great for development. Never use it in production because it takes a lot of memory and consumes a lot of the resources of the machine. But it's great for the learn so you can interact quickly with your with your own application. Now let's say that we want to also. I was forgetting something. So this item ID what we receive in the code, this is actually an integer because we are first, because we are annotating it with this column int. Then the editor is going to know what is the type of this parameter. So we can say item ID dot and we're going to get autocompletion for integers. We're not going to get out of completion for strings or for something else. The editor knows that this is going to be an integer. And if we try to do some invalid operation like summing an integer with a string, then the editor is going to be able to tell us, hey, you are doing something that is not valid. You can sum an integer and the literal string sdf. So the editor is going to help us a lot just by using the type annotations. But FAST API is also going to make sure that we actually receive the data that we were expecting. So if we instead go, instead of item seven, we go to items full and we go there, then we're going to receive an automatic error. The client is going to receive the client in the browser, it's going to receive an automatic error saying, hey, you have an error in the path. In the item id, which is a path parameter, the value is not a valid integer. So this is not a valid integer. So the fast API code here is actually in this line here is protecting us from running code with invalid data. So the data that we receive is going to be valid and it's going to be checked and converted by fast API. Also notice that this is a URL. If we put that 7 here, it's part of the URL. And a URL is just text. So fast API is also doing automatic data conversion because this is naturally a text. URLs don't have a data type. So fast API is converting the data to extract that information. Now the coolest part I guess of Fast API is that if you go we have been just like interacting with this, typing the URL, but if you go to slash docs, then you get this nice automatic documentation user interface for your API. You can see that this is based on our own code. So we have here is the main get that just returns hello world and here's the get items item id and we can actually try it out. We can type something here, we can type seven and then we execute it. And when we get the response from our server that is running here and we get the response here with the data item ID message hello world. Actually we don't need this message hello world anymore. This was just for the demo, so we can just save it again. This is reloading. We send the request again and now we get the new response without that message. So we can interact live with the API that we are building and that can help a lot. Now let's actually send somebody to this request. So Fast API is built on top of two libraries. One is Starlet and the other one is Pydantic. So Starlet does all the web parts and Pydantic does all the data parts. You normally don't interact with Starlet directly, but you interact a lot with Pydantic directly. And Pydantic is even great even if you're not building web applications just for general data validation. So we're going to import from Pydantic, we're going to import base model and then we are going to create a class inheriting from that base model. So we're going to create a class item that inherits from base model. This is going to have a name that is a string, a price that is a float. We don't want this offer, we want something. We want tags. We want to have a tags that is going to be a list of strings that by default is just an empty list. So we are just defining what is the data shape that we expect to receive. And now we can use this class in a new path operation. I'm calling this a path operation because it's a combination of a path like this one and an HTTP operation like this one. You can also call it a root or an endpoint or however you prefer. But because in FAST API you normally have the two things together, the path and the operation. I'm calling it a path operation. So now we're going to create a new one. This is going to be app post. Now this is a post one. This is whenever we send an HTTP post operation. And this is going to say I want to create an item and this item is going to be this parameter item is going to be an instance of the class item. This is standard Python type annotations to do that. Now I can save this. This is reloading automatically, can refresh the documentation page. And I can see that now I have the post here that takes the items and I even have the automatic example showing the data with the same shape that we defined it. Here we have the name that is a string, price is an integer. Well, actually a float and tags is going to be a list of strings if I send some data. So full bar, whatever, execute that, I'm going to receive the response here. Now if, let's say I remove the tags, I just send this like this. This is still going to work because the tags have a default value of an empty string. So this is still working. But if I remove the price, I try to execute it, then this is also fastik is going to tell us exactly where the problem is inside of the body, the price field. This was a field that was required and you didn't send it. So it's telling us it's doing the automatic data validation based on the data shapes that we define, so we can define arbitrarily nested data structures however we need. And FAST API is going to do the validation of that for us, even if we do deeply nested data structure. So for example, we have the tags that are supposed to be a list of strings, or in terms of JSON, it will be an array of strings. And let's say that instead of passing a string, we pass a dictation object or the equivalent is a Python dictionary. So this is going to be named full and we try to execute this API is also going to do the data validation for that. It's going to tell us inside of the body, inside of the ties at index one. So index zero will be this string and index here is fine, but index one is the one that has a problem, index one. A string type was expected. So we get all the data validation even for deeply nested data structures. Nice, that's very cool. So this is quick overview of FAST API in general, of the kinds of things that you might need now we're going to actually do the machine learning part. So now we're going to get a app get. We're going to get a generated model. So let's say generate, generate. No. Oh, I told you that you were going to see types, typos, generate. No, no prompt yet. We're going to generate an image here and we're going to receive a prompt. Now we are saving this. Let me reload this page. And you can see that we have here the new endpoint or the new path operation generate. And now this prompt, because this prompt is not part of the path here. This is going to be read from the query parameter. You can see that this goes from the query parameters. So I'm going to try it out. Hello, if I execute it, you can see the query parameters are these things that go after the question mark. So we are telling it, hey, I want the query parameter called prompt to have a value of hello. So we are sending the query parameters here and that's how we just define it. If it's not part of the path, then fast API is going extracted from the query parameters. Now we just got the prompt, we just need to return the actual image. So what we are going to do is that we are going to. I'm going to stop Ubicorn because I don't want it to be reloading every time and I'm going to import from ML import obtain image. I don't want Ubicorn to be reloading because whenever this line is run from ML import something, then this whole file is going to be executed. This is going to execute the creation of the function. It's not going to execute the function itself. It's not going to call the function. So this line of generating the actual image, this is not going to be called for. The creating creation of the function is going to be called the creation of the function. Doesn't matter. But loading the model is going to be quite slow because the model is slow. So we don't want Ubicorn to be reloading the file every time we save something and then just like taking a long time to reload again and reload again and reload again. So that's why I stopped Ubicorn and remove the result. Now what I'm going to do is that I'm going to use that obtain image prompt and I'm going to say, hey, I want to have an image that is going to be equals to obtain image of the prompt that I received in query parameters and I'm going to have the Number of inferences, that is going to be five. This is just the same small number of inference steps. And we're going to have the seed, which is the same fixed one. Just to make sure that everything is working, we want to have like a simple version first. And then what I'm going to do is that I'm going to save this image. So image dot save. And then I'm going to save it in a file image png and then I'm not going to return a JSON dictionary, I'm going to return a file. So I'm going to. From fastapi. Fastapi. Those requests. I'm gonna. From the first. No, the responses. Yeah, I'm gonna import a file response and then here I'm gonna return, return a file response with the same image. And that's gonna do it. Now I save it. I can restart Ubicon without reload and, And you can see that Ubicon is taking a little while to load because it's loading the whole machine learning model now it has to load the whole machine learning model whenever it reaches this point in the code. Now this is going to finish at some point. Then we're going to be able to reload here. If I refresh the page, this is not going to run because everyone hasn't finished it. And then. Okay, this is done. Nice. Amazing. Now we can generate it. So let's try it out. Let's write the prompt and the prompt is going to be a photograph of an Ostrom riding a horse. Let's copy this prompt, let's generate that, let's execute that. And you can see that this is using the device cooler. It's generating the image, it's taking a while. And then it's generating the image, saving that to a file and then returning us the file. And we have the nice random astronaut on the horse. This is not the coolest one. Let's actually build the coolest version. So we have this thing to generate the image, but we have a bunch of parameters that we can receive and we want to pass those parameters here. These parameters are taken from here. We can actually copy all these parameters just as is. We can just copy, paste that into this function. We can just copy, paste them and actually let me format the code. And now we can use all those parameters inside of our function, the function that we created. Let me format the code again so that we can see that. So we take the seed, we take the non inference steps, we take the guidance scale. And because all these parameters are just type Annotations, FAST API is going to be able to extract all that information from there. So now we're going to save it, we're going to stop uvicon, we're going to reload. Let's start UVCON again. This is going to take a little bit to load the model again and then we're going to be able to use, use this code. So let's see, let's see what we are doing. We are taking the same parameters. So seed is going to be an integer that can also be none by default, is none because it has a default, then this seed is not going to be required. If we take the normal inference steps, this is an integer. So Fast API is going to validate that we actually send an integer or that we don't send anything because this has a default value of 50. Let's actually refresh this page because this is already done. And you can see that now we have all the parameters here. So you can just try it out. Let's try it out with the same prompt that we were using. Photograph of an astronaut riding a horse. Give me that. And we just execute that. We're gonna execute it, we're gonna get. This is the real thing, so just brace yourself. This is the cool astronaut riding a horse from a web API that we are building. This is just taking a while because, you know, this is a massive machine learning model that is burning my laptop and it's probably gonna lift off before we finish the streaming. So this is processing that. It's gonna generate a very cool osma. While that, while this does that, let's actually see what are the things that we have here. So we have the seed, we could put the seed here to fix the type of astronaut that we receive. We have the number of inference steps that has a default value of 50. We have the guidance scale that has a default value of 7.25. We could change it or we can just leave it or just not even send it. And then FAST API will use the default values because this has all type annotations. Then FAST API is going to do automatic data validation, data conversion, all that stuff. We just use the same type parameters, the same type annotations that we had for the function and we just copy paste that and then we just use it here. And now we automatically have data validation, data documentation here, and data serialization. So this is almost done. And now we're gonna see what did we get? Okay, this is a nice strange black and white astronaut riding a horse in a very. I don't Know, kind of a western black and white style. Nice. Ta da. We got the nice AI machine learning API and you saw that we just like built all these very quickly. I hope so, without so many mistakes. Nice. Now let's go back to the slides for a little second. So we were talking about the code. I wanted to tell you a little bit just very quickly about what we were doing. We were saving the file. Actually, let me show you the code again. We were saving the file here and then we were loading the file again. This means that if we have a client that is sending some request and then we have another client sending another request, we might end up having the same file overwritten by one and the other and we don't want to send the file for one request to the other. We could have then random names for the files, but that will also mean that we will be reading and writing to disk, even though we actually don't need to read and write to disk. Normally reading and writing to disk is a lot slower than doing it from memory. So we could want to do it from memory. In this particular case it wouldn't make that much of a difference because the machine learning model is just taking so long because it's so heavy that writing or not writing to this is going to be negligible. But if we had like a faster machine learning model or something like that, we would probably want to avoid reading writing from disk. And what we could do will be to import a streaming response and then we could import IO, which is this standard library module. And then we can create a memory stream which is kind of a fake file in memory. And then we can save the image to this memory string in the format png. Because this doesn't have an extension, we have to tell it explicitly. Hey, this is png. They will tell the memory file object. Hey, go to the beginning of this file because we want to send the whole file, not the end that doesn't have anything. And then we will send that in the streaming response class here. Also telling it the media type is an image PNG because again, this doesn't have an extension so it cannot infer what is the type of GPG the object. But like, yeah, anyway, this is just like the. A little bit more for all sort of version where we don't even read and write from disk. We just like do everything in memory. Nice. So a little bit, a little bit more about FAST API. What's this thing, this thing about performance? So this is a benchmark run by a third party called taking power they run out. Compare a bunch of different languages and a bunch of different frameworks. These ones that I'm showing you are Python and Go Lang Go are the green lines, Python are the blue lines, which Go is a compiled language. So Go has to be faster than Python. Nevertheless, Fast API and a bunch of others get to have performance that is comparable to at least some of the Go frameworks. So this is quite nice. And this is while staying on the comfortableness of Python, where we can do all the machine learning and cool stuff. Which one is the fastest is not necessarily that clear. Ubicorn is the server that we were using to run FAST API. Ubicorn is a framework, it's a server to run the thing. So ubicorn will have to be on top of everything because ubicorn is running everything below. But anyway, it has just some random noise. The cool thing is that it has 15,000 requests or something, so that means it's over 9,000 requests per second. The thing is that you can get like kind of the best on the ballpark of the best performance that you can get in Python while staying in the comfort of Python and getting like all these additional features about data validation, documentation, serialization and all the stuff. Fast API also has a bunch of additional features that you can use. Dependency Injection, Security, OAuth2 integration, WebSockets, Files, background files, tasks, templates. Like all the things that you will expect from one of these frameworks. It's just that Fast API is optimized for building web APIs, which is the main type of thing that we are doing with web frameworks lately. And that's it. Thank you very much. I think we can probably jump to the questions.

Wow, Sebastian, that was fast and totally awesome. Thank you for walking us through that. And you know, I think we've got a nice chunk of time to get at some of these questions. Thank you everyone in the audience for asking those questions. Feel free to go into slido, keep upvoting, keep asking questions that you're interested in hearing from Sebastian about. So yeah, wow. Very, very cool stuff. The first question we've got that is by far the most upvoted is can we have a link to the repo with a template project? And Sebastian, I think we, we've actually started working on that on our side already. So we will definitely, I think.

Yeah, yeah, absolutely. We will, we will, we will share the code afterwards, dude, like make sure to follow deep learning AI and for brain, because they are going to share like the final code of this whole thing that we were playing with. But like, yeah, absolutely.

Yes, yes. And you know, come back, rewatch the video on half speed and try to live code it again because yeah, that was, that was awesome. Okay, so the next question is what deployment options for Fast API web app are available? Is there any difference in the way the app needs to be packaged for serverless versus virtual machine? For example, AWS Lambda versus AWS EC2?

Yeah. So if you are running the thing yourself, you're running a command directly as it will be running on EC2, running on a Docker container, or running directly on a machine or on Heroku or something like that, where you can tell which is the specific command. You just tell, hey, I want to run the command UVicorn and you tell Ubicorn. Why is the application that you want to run so fast API is built on top of a standard called asgi. This is a standard for Python for building Async web applications, Async web frameworks actually, and the communication between the server thing that is ubicorn and the framework thing that is Fast API. So Fast API is based on this standard. So you can just like use the, the common techniques that you will do. If you want to run on a serverless system like AWS Lambda, then you will use another application that, another package that is called Mango that just wraps Fast API to make it compatible with the specifics of AWS Lambda. If you are doing that on Google Cloud, that will be Google Cloud run, which is those Docker containers. So you just delete to run ubicorn directly. And also Azure is also building a bunch of things to support directly as the applications and to have very good support for Fast API. The same with all these other providers. Most of these providers have a way to support one of these mechanisms to run it, but normally it's very simple. Normally you just run uvcon with the app directly and that's it. Very cool.

Okay, the next question we've got is about scaling. So can you talk a little bit about best practices for scaling up, handling, you know, hundreds, thousands of requests happening in parallel?

Yeah, so, well, first I recently, well, like some months ago, I revamped all the documentation in Fast API about deployment and like what are the actual core concepts that you need to have? It's actually even independent of Fast API that you need to take care and think about requests per second. What is the request, what is latency? You need to take care about memory and CPU and like the ports and like what are the things that you're actually thinking of then what are the mechanisms to deploy? How do you deploy that if you do it manually, directly? How do you do that if you do it in containers? All the stuff is like quite long. And I won't be able to say everything here, but in general you are normally running some type of system that is normally scaling. You're normally running like, I don't know, some type of containers or some third party tool that is scaling your application automatically. So if it's lambda, then they are going to take care of of run multiple instances of the same thing. If it's kubernetes or something like that, then you're going to have containers that you're replicating outside of the process. So you will normally have a single Ubicorn program running a single fast ID application without workers, without any wrappers, just that directly and then you will replicate that container multiple times. This on the on the horizontal scaling to be able to have many fast API applications that have your same code all running at the same time and being able to handle like any, any load that you need to send it. Now about having a squeezing performance of a single fast API application, the thing is the recommendation will be just write your normal code. That's one of the things that I probably didn't mention. Fast API is actually an Async framework underneath. So if you write async, wait, sorry, as you can await functions, then fast API is going to be able to run them and like do all the high performance stuff. But you don't have to do that. You can write normal functions and fast API is going to take care of do the smart thing to not do, to not damage performance. The technical terms are to not block the main event loop on the main thread. If you don't understand that, don't worry. I don't fully understand that. But the thing is you can use async and weight functions or normal functions. If you use async and await functions, you need to make sure that all the packages that you use inside of those functions are actually compatible with Async and await so that you are not blocking that. For example reading a file and then it doesn't have an await, then Python is just going to be waiting for that instead of just going and handling another request at the same time. That's the whole benefit of Async and Await is that Python can handle many requests during the same period of time. If you need to squeeze performance, then you can just make sure that you use async and wait and that the libraries that you use are compatible with that. Then if you need to squeeze more performance, you can avoid having Fast API automatically serialize your data because in FAST API you can return kind of any type of object. Sometimes you can even return like objects that come from the database. And fast API will do the best job it can to convert that to JSON. But if you want to squeeze performance, you can make sure that you convert it to JSON directly and that you return a response directly. And you can instead of using the standard JSON response, you can use JSON response using or JSON or JSON which is a package that serialization and is written in rust so has much better performance and it's part of the, part of the first API included responses that you can use. So there are a couple of tricks that you can do to squeeze performance as much as possible. But the first thing that you will probably want to do is just write the simplest code you can and just scale horizontally, just add more nodes, just like add more containers to that and you're going to be like, you're going to get far like far away with just that before having to scroll squeeze performance in the code itself. Like you're going to have clients and you're going to have income before you need to go and squeeze the actual code internally. So like, yeah, if you're just building something, just write standard code and whenever you need to increase the throughput, then you can just like scale horizontally and just add more nodes or more containers.

Love it, love it. Yeah, wait until you have clients and wait until you have some revenue coming in to worry about scaling. So I want to follow up double clicking in on this sort of async development. Marcio Olivier Oliveira asks what is the best approach to retrain the model we have trained outside our application and also what is the best way to load them asynchronously?

That's like, the thing is that's something that there's, there's no, oh, this is the way. There's still no, no perfect way. It depends a lot on the team, it depends a lot on the approach, it depends a lot on how you are doing things. So for some people it makes sense to use a third party service that just serves machine learning models for some particular frameworks with some particular layers and they are compatible with that and then you are fine with that and you can just load them all there and then you can have a fast API in the middle that will be kind of a proxy between that thing that is serving models and it's just like doing all the authentication and handling all the requests and just resigning that that helps for some people. I have seen a lot of people that what, what they do is that they need to be retraining quickly and they need to have a bunch of different models and they need to be able to have them isolated. So what they do is that they train the models and then they have them packaged like in whatever is the standard format for that model in a file. Then they create a Docker image with that machine learning model inside of the document image and then just load them all inside of the FAST API application and everything is an assignment single container. If you do it this way, then there's just like a couple of things that you need to make sure you do and you don't do. So you should not load the machine learning model per request because loading a machine learning model normally means reading a file from disk, which is very large. Reading from disk is slow and if the file is large, it's going to be super slow. And if you do it per request, that's going to be super, super slow. One thing that you can do is load the model just like load the machine learning model at the top of the module, so at the top of the Python file. That will mean that whenever you import that file or whenever you start the FAST API application, that's going to take a while loading that machine learning model. The only caveat is that if you have that machine learning model in some particular file that is part of your whole application and you need to import that Python file from different places, then all those places are gonna end up. The first place that imports that file is gonna end up loading that whole machine learning application, that whole machine learning model, and that's gonna take a while. The super duper trick that you can use is actually use a utility from the standard library from the FUNK Tools library, the tools for functions. That is called an alert. You cache, which is a least recent used cache. You add the decorator on top of a function that will return the actual model that is loaded and then the next time you call that function, it's actually going to return the same object that it returned the first time. So it's just going to load them all the first time you call it, and then it's not going to call that again. That's kind of like the advanced trick that you can use if you need to optimize that type of thing. You can read more about that in the FAST API documentation about Settings is not related to models, but it's also reading from files. So it's kind of the same trick. You can do that if you need to handle all the different possible use cases. It's just that it's a little bit more advanced. But probably for the first use cases and for the first things that you do, you can just load the machine learning model at the top of some Python file and that will probably be the easiest and the best way for a long while to do.

Okay, okay. Stay away from loading it per request. Yeah, I love the very simple rules that we're getting out of this and obviously there's a ton of advanced things that we can do. Okay, the big question, here we go. Sebastian. Okay, Fast API versus Flask for building ML and AI web apps. What's your take? What's your hot take on this?

So like, I don't know, there's a lot of Fast API versus Flask and like, I think that the main attention, the main focus should actually be on like, hey, Flask tools goes like up to here. Flask has done a lot for us. Like there's just appreciation for Flask. If you go to the Fast API documentation, there's a section about inspiration and alternatives and like, there's just like a lot of inspiration that Fast API has taken from all these previous frameworks that have great ideas. And Fast API is just like trying to take all these ideas and put a bunch of these ideas together in the best way possible. So yeah, like Flask has taken us a very long way. The thing is with Fast API you get some extra advantages probably that are particularly applicable to modern code building APIs. If you are not building an API, if you are just rendering HTML on the backend, it's just gonna probably gonna be like kind of the same. It's not gonna, you're gonna not gonna see like, oh, this is much better than the others, like no more or less. But if you are building APIs, then fast API is probably going to be going to help a lot more because it has a lot of the things that you Normally need for APIs, like data validation, data documentation, data serialization. It's all based on standards, it's all based on these open API standards, which also means that you can generate clients automatically for any programming language because this is all based on the same open API standard. That's all the documentation for that Fast. Yeah, and you get is all based on the standard type annotations and is also Async and Await. These are for example, some of, some of the features of modern Python that were not even available when Flask was created. So Flask had no way to be designed to take advantage of that and exploit it. So it's like just some of the advantages of like, it's just like newer, so it's been able to take advantage of all these new things so that you can get like auto completion, inline errors and all this stuff everywhere inside of your code. But at the same time you get data validation, serialization documentation and all those things. You can probably see like there's a whole section of the FAST API features and there's a section about alternatives and inspiration and you can see like, also like what are several other frameworks and how FAST API learned some of the things from those frameworks to try to apply them again here in this way. The other thing is, if you are already used to working with Flask, then FAST API is going to be very similar because FAST API inherits a lot of the design from Flask, it learned a lot from Flask and actually recent versions of Flask copied some of the designs of Fast API. So like, I guess we just kind of cross pollinate with all these ideas. But the thing is, if you're already used to Flask, you can probably just read the tutorial on the first page of Fastapi. Not more, just that that's going to take like 10 or 20 minutes and that's going to be enough for you to decide like I like this thing or this, this doesn't make any sense. That's going to be enough for you to evaluate if you like it or not. So it's quite an easy investment. Just like 10, 20 minutes to see like do I like it or not?

Love it. Love it. Awesome. And you know, I think, you know, you even mentioned when we were, when we were kind of doing a run through, you know, Automatic Docs. The best part and I think this is something that we've seen a lot in our, and you know, our courses at 4th Brain is that's what people love a lot about Fast API is the Automatic Docs. I don't know if you had anything else to add to that, but that's certainly what I've noticed.

No, yeah, me too. Like, you know, like I built this thing to just to satisfy myself. I was using Flask and I just wanted some extra things. So like I wanted to have those that I enjoyed that a lot. But like, yeah, you know, like going back to the question, like I have been hearing more and more teams impact even more in machine learning, I guess because machine learning is in many cases just APIs and like not even much more. I have been hearing like more and more that the teams tell me like whenever we're building something new it's just like it's like by default directly with fast API. So like we have previous tools and previous parts that are built with Flask, but we are building something new then that goes directly to fast API because like that works better for them or they also migrate some of the things from Flask to Fast API. But like yeah, of course there are many Flask applications already deployed and like they are just like working fine. But in many cases when you're building something new and you want to take those advantages like it probably makes sense and I actually do like the docs a lot. Like I'm probably not a very good programmer. I need to have auto completion and inline errors and I need to be able to have like the browser telling me what are the parameters that I can set. Oh right, it was his name and like all that. So I need to have all the help I can get and I built API to give me all the help I could get because I need the help from the tools.

Yes, yes. Awesome. We have one or two more questions here. We're going to get super tactical with this one. This is highly upvoted and it's a common point of confusion talking about Anaconda versus Virtual environment directly in Python it said your command line says you're using Anaconda. Is there a reason why you're also using Virtual Environment VN and PIP at the same time?

Yeah, no, these. So just for, for this tutorial and for what we saw, disregard the fact that it's Anaconda or not. It's just because this is the old installation I already had. So like it's just like already there. But I'm not using any of the components of Anaconda itself, it's just that I had that installation but I'm not using any of the internal parts of Anaconda. If you are building a project like it will probably depend on a lot of the projects. So Anaconda, Anaconda in particular the package manager from Anaconda has some particular benefits when building tools and when installing pre built binary tools when those are available in conda Forge in particular, for example, if you're in a new architecture like one of the new Macs, an M1 in many cases they don't have a pre compiled wheels which is the standard format for packages. They don't have them available. So but there some version available for conda so you can install those with conda in those particular cases is probably like a very useful tool or when you have dependencies that go outside of Python itself or the Python ecosystem, if you have R dependencies or things like that. But in most of the cases you are probably a. You're probably good enough and probably better off with just like playing Python and using a visual environments or pip, or even if you use a tool that helps manage virtual environments like Poetry or something like that, but that is still in the same ecosystem of a PIP that installs from PYPI and that installs with the same PIP mechanisms and all the stuff. Because you get like the latest versions of Python and the latest official versions of those packages and like all the stuff. But in some advanced cases, like you will probably end up needing to use Anaconda or conda or if you need to have like some environment that is just like pre installed with everything and you are just using them, that you're not really installing packages or things like that. In some of those cases it might, it might work, but like yeah, if you are just like building a project, it probably makes sense just to use virtual environments in some way. It could be what I was showing you. I was just showing you the simplest case, which is with the tools that come with Python. But you could use some tool that helps you manage all that at the same time, like dependencies, virtual environment and installation. With for example, something like Poetry that helps you do everything on the same package. Then there are some other alternatives, but.

Like yeah, love it, love it. Yeah. When in doubt, always just keep it as simple as possible. Sebastian, wow, I've learned so much today just listening to you. I cannot believe how much content and questions that we were able to get through today. Thank you everybody for your questions and Sebastian, thank you again so much for your amazing presentation and answering. Really everything that the audience had to throw at you.

Thank you. Thank you very much for inviting me and thank you everyone for staying. I hope that was fun and enjoyable. It's a massive pleasure to be here with you all with Forgoing and deep learning the. That's so cool.

Yes, absolutely, absolutely. This is, this has been amazing. Very cool and this bright.