LARRUZ shows Draw-A-UI, an open-source tool that converts hand-drawn sketches into working HTML apps using GPT-4 Vision. Draw a clock or calculator on your screen, and it generates a single HTML file with Tailwind CSS. Community examples include a terminal, Snake game, and physics simulator.
Questions this video answers
LARRUZ
November 18, 2023
Takes current canvas SVG or screenshot → converts to PNG → sends image to GPT-4 with vision with instruction to return a single HTML file (uses Tailwind).
Yes — users must provide their own OpenAI API key to use the service.
No — the narrator cites a disclaimer: it is a demo not intended for production and lacks authentication.
Chapters
This will blow your mind and it's probably one of the coolest projects that I have seen in a while. It's an open source project called draw a UI and it uses GPT4 with vision to generate HTML code based on the wireframe you draw. So it's very similar to one of the earlier demos of GPT4, but it's a lot cooler. So I'll show you how to get this up and running, but before then, let's see how it actually works. So it takes the current canvas SVG or screenshot, then converts it to a PNG and then sends it to GPT4 with vision, with instruction to return a single HTML file. With tailwind, you actually get fully functional app out of this. Now, before you start thinking of putting this in production, you need to read this disclaimer which states, this is a demo, is not intended for production use. It doesn't have any authentication, so you will go broke if you deploy this. So keep that in mind. Now, you can run this locally if you want, but I would recommend to use their website. And it's a pretty nice interface. So let me show you how it works and then I'm going to show you some really cool examples that people have built on top of it. So at the bottom you see this toolbar. So let's draw a very simple interface. So what I'm going to do is I'm going to click on this up arrow and then select a circle and let's draw a simple circle. Next we're going to add some numbers to the circle. I'm going to click on this. So I think you know where I'm going with this. So I drew the circle, put numbers on it, and then drew these minute, hour and seconds hand using this arrow. So now we have a user interface of a clock. Let's see if we can convert this into fully working clock using this amazing tool. Now, you need to provide your own OpenAI API key, so you can paste it in here. Now, to convert this to working code, we will need to select the whole clock and then we're going to click this button, make real, and we'll wait for the magic to happen. So the process takes a few seconds, but we have a fully functional clock, which is absolutely amazing. But that's not the best plan. You can actually integrate this into your own websites. So what you can do is you can click on this small copy button and this will basically copy the code. This is an online HTML editor. So I'm going to paste the code that we copied in there and let's click Run and you can see the fully functional clock, which is absolutely amazing. So you can now simply draw user interfaces and this tool will convert it into fully functional widgets and websites. This is mind blowing. Now the best part is this is not the only one thing you can do. You can do a lot more complex interfaces. So let me show you a few examples. In the rest of the video, I'll show you a few amazing examples. So the first one is actually from the developers themselves and they simply drew an interface of a terminal, and GPT4 with vision is able to convert this into a fully functional terminal, which is crazy. And you can embed this in your websites if you want, because this is fully functional HTML code. Now you can give it instructions as well. So for example, here, this user named Aiden, he drew this cube and then he's giving the instruction make a make rotating cube out of it. And then GPT4 with vision is able to just look at the image and the instruction and is able to create an animation out of it. This is pretty awesome. Here's another cool example from you. It's a fully functional calculator. So you simply need to draw the user interface. Now the amazing thing is that GPT4 with vision can understand what you're drawing and then based on that, it adds all the functionality in there. So if you see the user is not providing any instructions, so we simply draw a user interface and then the GPT4 with vision is able to make deductions based on what it's seeing and what the functionality is supposed to look like. I think this will really change the way we develop software and tools now. So here's an example of a fully functional drawing app from olic. Now you simply need to draw the user interface and GPT4 with vision is able to deduce what type of functionality you want. Just wait for a couple of seconds and it will come up with a fully functional UI and app, which is pretty awesome. So in this case, as you can see, it created this drawing app. There is a pencil tool button. If you click on this now you can start drawing, which is mind blowing. Like it took under a minute to create this. I personally really like this one. This is from a user, the one always wiser. So basically he's describing a physical phenomena and GPT4 is looking at both the instructions that are provided as well as the images. And it's able to replicate that physical phenomena along with all the physics that is happening, happening in the background. So I think this will also really help teachers or educators to build some really cool animations in order to convey different ideas. And here is one of my favorites. So Austin actually drew this user interface of the classic Snake game. And now by just looking at the user interface that is drawn, GPT4 with vision is able to create a fully functional classic Snake game out of it. And you can actually play this, which is pretty awesome. It's actually really exciting to see what people are able to build with GPT4 with Vision API. And it's been just under 10 days, so this is pretty crazy. So there are already people building on top of it. So here's an example from Lynching, who created another app which is creating image diffusion models based on the drawing that you see in here. So somebody is drawing a mountain and then using the diffusion models that is being converted in near real time to a scene and you can see that image being created in the right bottom corner. So this is pretty brilliant because you're creating apps just by drawing them and adding functionalities that required a lot of engineering and code before this. One thing which I would like to highlight is that this is the first iteration of Draw A ui. So although it's really amazing, but still there are bugs. So sometimes you have to change your UI a little bit in order for it to be fully working. So keep that in mind if you are experimenting with it. But even though I think it's still a really amazing idea, another thing to keep in mind is that it will cost a lot of tokens. So make sure you keep an eye on your API usage because this can add up really quickly. Now I think it's possible to replace the multimodal GPT4 Vision model with an open source multimodal model such as Lava, because it has the ability to understand images. So probably it's possible to create a tool like this using completely open source large language models. Based on the progress that we have seen in the open source community, I feel like that is going to happen pretty soon. I'll recommend everybody to check out this project and give it a star. It's absolutely amazing. Now you can run this locally as well, so there are instructions on how to install it. But let me know if you want me to make another video on local installation of this project. I can't wait to see what's next for this. I hope you found this video useful. If you like the content, consider liking the video and subscribe to the channel. Thanks for watching and as always, see you in the next one.
