Run SDXL Locally With ComfyUI (2024 Stable Diffusion Guide)

So it's been 10 months since I've made a stable diffusion tutorial, and in the world of AI, 10 months is like forever. So, so much has changed. There's been better tools that have come out, better processes to run stable diffusion, and to be honest, it's actually gotten quite a bit easier to use Stable diffusion. In this last video 10 months ago, I showed you how to set up stable diffusion using the best tool at the time, which was automatic 1111. In this video I I'm going to show you how to set it up using ComfyUI, which most people who are using Stable diffusion these days seem to find it a better user interface. First, I'm going to quickly talk about why you might want to use Stable diffusion as well as why you might want to use ComfyUI. If you came to this video and you just want the ComfyUI installation tutorial, I did timestamp this video so you can jump straight to the installation process. So with tools like midjourney and Dall E3 and Google's Imagen and all of these AI art tools available, why would you want to use Stable Diffusion over the alternatives? Well, probably the reason most people choose to use Stable diffusion is that it is free and it is open source. It's also completely uncensored, so you can generate any images you can imagine, and I mean literally anything. I've accidentally created Nightmare Fuel in the past. Also, since you'll be installing stable diffusion locally on your own computer, you don't really have to worry about privacy concerns or, or companies collecting data from you, because you can use this without even being connected to the Internet if you want. And finally, stable diffusion is ultra customizable. You can use things like Control. Net to get the exact pose you want. And there's all sorts of other add ons that extend the capabilities of stable diffusion. So this is automatic 1111 here, this is what I've used for the last year and a half to generate images with stable diffusion. It does the job for, but there's definitely some downsides to it. Now this is what Comfy UI looks like. It's sort of this visual workflow where you can see the arrows pointing to the next step in the process. So you get this full visual of everything that's happening during the processing of your image. And once you've used it for a little while, you feel like you have a little bit more control over each step in the process. Now, other than the reasons that I just mentioned, ComfyUI is actually a lot easier. Easier of an install than automatic 1111. You'll notice once we get the process started, we'll have it set up and running within minutes. You can also save entire workflows using ComfyUI. So if you get a flow of the perfect prompt, the perfect negative prompt, the perfect model, the perfect GFC scale, and you have every little element dialed in the way you like it to make the style of images you want, you can save that entire workflow, reload it anytime you want, and just tweak the prompt to get exactly the same style again. And with ability to create and save these workflows, you can also use other people's workflows. Here on CIVIT AI we've got the Comfy Roll SDXL templates and you can see these are custom workflows, a simple, intermediate and advanced workflow that somebody else created that we can download and just use to jumpstart the process. Now, in order to get Comfy UI to work really well, you're probably going to want to be using a PC with either Windows or Linux installed, and you're going to get the best results. If you're using an Nvidia GPU now, you can have process through the cpu. It is just way slower when you do it that way. And if we look at the ComfyUI GitHub page, we can scroll all the way down here and you can see that there are instructions to use Apple Mac silicon aka an M1 or an M2 or now I guess an M3 and there are instructions. But I have read over on Reddit that a lot of people have had a lot of issues getting Comfy UI installed using these instructions. However, I got you covered. I did find this YouTube video from AI Animation called Install ComfyUI on Mac OS M1 or M2. He uses a tool called Homebrew to get it installed. So if you are a Mac user and you can't follow along to my PC installation instructions, I'll link this video below and you can follow along to his Mac install instructions. And then pretty much everything after it's installed is going to be completely the same. Cool. Now that you know why you might want to use Stable Diffusion, why you might want to use comfyUI and what specs you need to make it work. Let's walk through the install process. Like I mentioned, it's super easy, super quick. The longest portion of this entire process is just going to be you downloading the actual files that you need. So to install this, you're going to go to the GitHub page, which you can find over at GitHub.comfyanonymous/comfyui. Don't worry, I'll link up below so you can just click the link and come straight here. Once you're on this GitHub page, we're going to scroll down all the way until we get to the installing section. There's a big blue link here that says Direct link to download. You're going to go ahead and click that link and it will download the installation files. Once you've downloaded the installation file, we'll go ahead and paste it in the folder where we want to install ComfyUI. In my case, I'm installing it on my E drive here in a folder called AI. And then I'm going to create a new folder called ComfyUI. We'll jump into this new folder and we will paste the file we just downloaded into this folder. Now, you'll notice the file extension that we just downloaded is a dot 7Z file. That's a 7 zip file. I personally use a tool called WinZip on my PC to unzip files. So if you have WinZip installed on your computer, that should unzip it for you. There's also a tool called 7zip, which is what a 7. Z file was created with. If you go to 7-zip.org you can download the latest version here, install it, and that will help you unzip the file as well. Since I have WinZip already installed, I could go ahead and right click on this, come down to winz, unzip here and just click unzip to here and you can see it will start to extract the files. It is a large zip file with a lot of little files in it, so it might take a few minutes to unzip. All right, so it did take about five minutes or so to unzip this because it is a very large file. But once it's unzipped, you can go ahead and delete that 7. Z file. And now we have this new folder, comfy UI Windows Portable. Double click into this and you are presented with two options. Here you've got Run CPU and Run Nvidia gpu. If you do have an Nvidia GPU on your computer, you're going to want to use this version. If you're not using an Nvidia gpu, you're going to want to use the CPU version, which will be a lot slower, but it should work. Let's go ahead and double click on Run Nvidia gpu, you might get some sort of error saying that Windows is protecting your PC because it doesn't know what this file is. You can click more info and then run anyway, and then it will start the run process. Now, one of the beautiful things about this way of installing ComfyUI is, is you don't need to download Python or get or any of those tools, because this installation process takes care of all of that for you. And after about 30 seconds we're in comfy UI. It's open and we can move stuff around. And if you need everything a little more separated so you can follow the lines a little bit better, you could drag it out however you want. There's just one problem. We don't have any models installed yet. If I click queue prompt, you can see we get this error that the prompt output failed validation. That's because if we jump back over to where we installed ComfyUI, we come into our ComfyUI folder here and we go to Models and then click on Checkpoints. You can see it says put checkpoints here. We haven't actually downloaded any Stable Diffusion models. We only downloaded the front end user face to run Stable Diffusion. So now we need to put a model here and I recommend installing Stable Diffusion xl. This is, at the time of the recording, the newest and most powerful model available. In order to download Stable Diffusion xl, you're going to come to this hugging face URL up here, which of course I'm going to link in the description so you don't have to remember this or type it all in. You're going to click on files and versions and scroll down and download this SDXL base 1.0 safe tensors. Now, it is a 6.94 gigabyte file. Unless you've got ultra fast Internet, it could take a little bit to download. But go ahead and download this file by clicking the download button here. You're also going to want to download the SDXL 1.0 refiner model card. This is part of the SDXL process. And again I will link the Hugging face page below. You're going to click on files and versions, scroll down and we're going to download the SDXL Refiner 1.0 safe tensors again. 6.08 gigabytes. Keeping track. We're up to over 12 gigabytes of file size for our Stable Diffusion checkpoints here. Thirdly, you want to download the SDXL VAE file. I'll link it up below, but go ahead and download this one here. SDXL VAE Safe Tensors. Go ahead and download that. I've already downloaded all three. Now I just need to move them to the proper places so that comfy UI can find them. So we'll start with the SDXL base 1.0 safe tensors that we downloaded. This is our base model. To go ahead and cut it from my downloads folder here, jump back over to where we installed ComfyUI, click into ComfyUI, look into models, look into checkpoints, and we will paste the checkpoint here. All right, let's jump back to our downloads. Now we have our refiner. Let's go ahead and cut the refiner from the downloads folder. Jump back to our checkbox, paste this in the same folder here. Cool. So now I've got my base and my refiner both in my Models Checkpoints folder inside of comfyUI. Now we've got our VAE file, so let's go ahead and cut this. I'm going to hit back on checkpoints. But down here we've got a folder for vae. Let's go ahead and double click into this. You can see it says put VAE here. So we'll go ahead and paste that there. And now we should be ready to run SDXL through ComfyUI. So let's jump back over to our browser where this Comfy UI tab is open. And let's go ahead and refresh it. And you can see down here under checkpoints, we now have SDXL base safe tensors. And if I click to the right, we've also got the refiner. But we're going to go ahead and leave it on SDXL base. So now it's going to load this SDXL base checkpoint here, this top box that says clip text encode. This is our prompt. So the default prompt that's here is beautiful scenery, nature, glass, bottle, landscape, purple galaxy, bottle. And then this one right here, it says clip text encode prompt. This is our negative prompt here. So anything we don't want, text or watermark is what they've got by default. Down here it says empty latent image. This is where we can set our width and height and our batch size. So if we want it to generate four images at a time, we can set the batch size to four. We just wanted to generate one, we'll keep it at one. This is basically the starting point for our stable diffusion image. We've got our seed here. If you leave this seed and the control after generate at randomize, it will create a new random seed every time you generate. You've got your steps here. I usually like to go between 30 and 40 steps. This is just kind of how long it sort of bakes the image for. I guess you've got your CFG here by default is set at 8.0. This is sort of how closely it's going to follow the prompt that you gave here. But it does seem that a number between the 7 and 10 range seems to be about that sweet spot. But it is something that you can play around with. For sampler name we've got Euler, Euler A and a whole bunch of other samplers that you can play with. I'm going to go ahead and just leave this on the default Euler. You have some alternate schedulers here. These are other things that you can play with and see how they affect your image. And then you've got your denoiser here. I'm going to go ahead and leave this just on the default of one for now. And then over here this is how it's going to save your image. So if I click into this I can actually change the name. Right now it's just going to name it Comfy UI and if I click Queue prompt we can see that it will run through all the processes and then generate an image for me. Right click on this and click Open Image. You can see here is the image saved to my computer at 512. By 512, however, there's a way to make ComfyUI even more powerful. You can add more of these nodes and have it do more stuff. Now the best way, the easiest way to add more nodes is to install this thing called comfyui manager. You can find it@GitHub.com well this URL here, it'll be linked up in the description. We're going to go ahead and right click inside this folder. Click on Open in Terminal and then you can see inside of the terminal it already opened to the exact right path for us. By right click it will paste in this get clone that we just copied from our GitHub page. I hit enter, it will then proceed to install the ComfyUI manager. I can go ahead and close out of this Terminal window here. You can see I now have a Comfy UI Manager folder available inside of my custom nodes folder here. Now if I jump back over to ComfyUI, come up to my refresh in the top left up here you can see it now added a Manager button that wasn't here before. I click into this Manager button. We have a whole bunch of New options. Most importantly, we have a button to install custom nodes. Now, a node is essentially each little section here our load checkpoint, our clip text encoders, our empty latent image, our sampler, our decode, and our save image. These are all different nodes. So by installing more nodes, we can add more pieces to the process. Let's click back into our manager here. Install custom nodes. And for example, let's say we want to install control. Net. I can search for control. Net and you can see right here we have comfy UI's control. Net auxiliary preprocessors. I can go ahead and click install on that. Once installed, it's going to ask me to restart comfyUI. So if I click restart and then press OK next to reboot server, it's going to go ahead and install control. Net and then restart for us. Now that ComfyUI has rebooted, if I right click anywhere, you can see I've got a menu for add node. I come down here, you can see I now have ControlNet pre processors. And now that we have the ControlNet node installed, we also would need the control. NET model. So if I come to my manager here again click on install models. Let's do a quick search for control. Net. You can see we've got different control. Net models that we can install here. And for this example, let's go ahead and install open pose. Since we're running sdxl, I'll grab the open pose model for SDXL here. Now that it's installed, I can close this, we'll close our manager, Click refresh. And now if we want to add control. Net into the mix, you come up here, add a node. Let's add an image load image node. So this is going to be the image we're going to use for our control. Net where it's going to try to find the pose of it and we can add a node. Come down to our control. Net preprocessors. Let's click on faces and pose estimators and open pose. And now we can bring this image into the image of open Pose. Let it detect the hands, the body, the face. You can turn off any of these if you want. And then if we want to preview what this open pose is actually doing, I can add another node. I'm down to image and preview the image. Now if I connect this image here. Now if I pull in an image like this one right here, and we cue our prompt, it should be able to find the bottom the pose. From this prompt here, you can see it found the face and the way the hands are positioned in this prompt. Now we want to make sure this pose is fed down into our image generation here so that it follows the same pose. So in order to do that, we need to add another node and to quickly search for a node. If I just double click over here, it brings up this search box. If I type control, you can see there's this control. Net apply. So I need to select control. Net app apply. And you can see down here where my text prompt is. I've got a line going from conditioning into our positive prompt. Let's go ahead and change our conditioning and pull it into our control. Net up here. And then this side, the conditioning output, we're going to bring it back around and pull it into our positive here. So now it's going through the text prompt, passing through this control. Net and then going back into our sampler. And then for our image input on the apply control. Net, we're going to pull it in from our open pose pose here. So I'm going to pull this little image right here down to our image box here. Now we have one box left, which is our control. Net where it's actually asking again which control. Net we're using. Now, to get this control. Net, I need to add one more node. So I'm going to double click right here. I'm going to type control. Net. We're going to grab a control. Net loader. And since the open pose model is the only model we've got right now, we're going to go ahead and keep that one selected. Pull our control. Net right in here. And I know it looks complicated, but if you follow the instructions I just showed you, you should be able to pull in this pose from this person into our control. Net image here. And let's just put an excited man wearing a colorful shirt on a beach. And I know this is starting to look more like a spaghetti bowl, but hopefully you're following along so far. If I hit cue prompt, it's going to load our control. Net models and our poses. And you can see down with our image here. It sort of modeled him. I think it's having a hard time telling that it's got two hands up in the air based on our pose here, but it modeled our image. And if I cue the prompt again, you'll notice that the pose is going to kind of stay the same every single time we've got this consistent pose. Now, one of the things that makes stable diffusion so amazing is that you can go to a site like Civit AI, go to the Models page and there are literally thousands of available models that other people have custom trained that you can use in your Comfy ui. Be advised, there are quite a bit of not safe for work models in here, and if you want to make sure you don't see them, make sure that up here you have blur mature content turned on. Otherwise you will see a lot of mature content on Civit AI. Now, the most popular model on Civitai is this Juggernaut XL. If you wanted to use this model inside of ComfyUI, simply click into it on Civit AI, come to your download button and download the model model Safe Tensor once again over 6 gigabytes, so it could take a minute to download. But once you've got the model downloaded, come to your Comfy UI folder, go to your Models folder, your Checkpoints folder, and drop the Juggernaut XL Safe Tensor file right here. And now you'll have the Juggernaut Safe Tensor model available under your load Checkpoint model. If you watch the video that I made about a year ago about how to inject yourself into an AI model and you created your own checkpoint with your own face in it, you can use that model inside of ComfyUI as well. Simply take the model that you created. It should be a file that ends in.ckpt and is probably going to be a long file name like this if you follow the Dream Booth model that I showed you. Now if I jump back to ComfyUI click refresh here you should see our model that we just loaded up as one of the available models. Again, it's really long, so it's taking up more than what's available in the box, but that's okay. So I'm gonna enter this prompt using my keyword to generate my face. Dominic Cooper Person Explain why that's the keyword in a whole bunch of past videos I want to put is excited and standing on a bright colorful beach this one. I'm also going to delete this node right here. That brings control. Net into our sampler and just bring our text prompt straight into our positive. The reason I'm doing that is because this open pose is designed for SDXL and not for this model that I created. I could install open pose for stable diffusion one point and that would work, but I don't want to get too into the weeds. I just want to show you that our custom model with our own trained faces should work. So let's go ahead and queue prompt and just like that it generated an image of me. It sort of ignored the part about standing on a bright, colorful beach. So if I up my CFG scale here a bit, let's throw it at 10. I'm also going to up the steps here to 40 and let's change our sampler to boiler a here, let's try this one more time. And there we go. There's me standing on a beach. So you can load your own custom models in using that method that I just showed you. And one final thing I want to show off. Other people can create workflows, and you can literally copy and paste their workflows into your Comfy UI and use them yourself. For example, here's this Comfy Roll SDXL templates. We can download this here and it basically gives us a bunch of JSON files. We've got two Advanced to Intermediate and Too simple. And if I jump to ComfyUI and I open the folder where I just downloaded these JSON files, if I want to use one of these templates then that this other person built, all I have to do is click on it, drag it, and drop it into ComfyUI. And just like that, it brings their entire workflow in. Now these are red because there's some extra custom nodes that I haven't installed yet that need to be installed. But once you have all the nodes installed, it's as easy as dragging and dropping it in and then it just works. So there you have it. There's the new way to install stable diffusion and get really customized with it and really, really dial in the control of what you can can do. Now this tutorial was a little more dense than some of my other tutorials. Hopefully you're able to follow along. If you want more Comfy UI tutorials and you want to go even deeper with it, let me know in the comments. I'm happy to make more advanced tutorials in the future. These types of videos don't normally do super well on my channel, but I like having them on my channel so that anybody searching for ComfyUI tutorials can find this in the future. And if you love nerding out about AI as much as I do, make sure you check out FutureTools IO. This is the site where I curate all of the coolest AI tools that I come across, as well as all of the latest AI news. And if you just want to know about the most important news of the week and just a handful of the coolest tools that I come across, make sure you join the free newsletter and I will send you just the important stuff to your inbox. It's totally free. You can find it over at FutureTools IO. And if you love AI tutorials, you love AI research, you love AI news, make sure you subscribe to this YouTube channel and like this video, and I'll make sure more videos like this show up in your YouTube feed. Thank you so much for tuning in. Hopefully you found this tutorial helpful. Hopefully by the time you see it, it's still relevant and still works. AI moves really, really fast, but there you have it. There's comfy ui. I'm having fun learning about it. Hopefully you enjoy it too, and I will see you in the next video. Bye Bye.