tldraw and AI on the canvas by Steve Ruiz

¡Ah! ¡Ah! ¡Ah! Craig— by the way, every time I come to Spain— uh, I have a Spanish last name but it's from Colombia so I say "Ruiz" and, uh... My Spanish friends always tell me I'm saying my own name wrong. No, no, no, it's Ruiz, Ruiz, something like that. It's Ruiz. Ruiz. Oh, no, something happened. Let me clean up my slide here. I am Steve from TealDraw, and you can find me on social media at SteveRuizOK. And this is TealDraw. So TealDraw is, uh, it's kind of a confusing story, so I'm going to do my best. Um, there's two parts to TealDraw. There is a developer SDK for people who want to build whiteboards or infinite canvases or anything like that. That is at tealdraw.dev. Um, React canvas SDK for React developers, infinite canvas, right? And then there's the thing that I'm showing here, which is tealdraw.com. This is supposed to just be a demo, but there's like 100,000 people who use it, so you can use it too. And you can use it for anything— diagrams, drawings. You can run your retro. It's collaborative. I'm not going to demo that on stage because it always goes the same way. But yeah, it is an infinite canvas with a lot of little details. Like, I am obsessed with arrows. And our arrows are really good. And there's a lot of juice. You could put arrows on top of lines and do spline-type lines. And the arrows should still kind of follow the spline. It's very, very, very fun. The ink— we have this kind of ink-style lines. I don't know if you can tell, but they kind of get thicker and thinner based on the density of the— the points and stuff, so like, you know, like, boop. I don't know. And part of my job as someone promoting and marketing and getting people to care about TealDraw is to make demos. It's an SDK. We can make a whiteboard out of it, but you can also make other stuff, right? So for example, we have like a PDF editor example, right? Where, you know, the camera is constrained. I can't go left and right, but I can zoom in and zoom out and, like, go down the pages and, you know, do my highlighting and stuff like that and export this. So that's a little demo. We have— actually, I have it with my own slides here. A demo where, like, we kind of mask everything that is not selected. So it's kind of a nice way to present content gradually. Maybe you have a classroom or something, you want to do it that way. And then other people build stuff with it. Some of those are our customers. People build things like— actually, this one just dropped today. It's like a— for, uh, ooh, social media. Uh, being able to kind of like construct these nice little feeds, um, add your logic to, uh, these like blocks like this, kind of like Scratch. This is TealDraw, you know, this is, uh, built with the SDK APIs and other parts that we baked into TealDraw. And now you have this really nice canvas experience that you could build without having to build the whole canvas. That's always the value proposition is like, if you have some idea that involves a canvas, don't spend the first 2 years and all your seed stage investment building just the canvas so that you can build your stuff on top of it. Use TealDrop, ship in a weekend. There you go. Demo's good. When AI came out, or at least when the language models got multimodal and they could start start working with images as well as text, we started making more and more demos that took advantage of that. The main reason we did that is because they went viral again and again and again. People like the idea of taking a drawing of a cat and turning it into slop. People love it. So I'll show some of those demos. The first one— actually came second, but I'll show it first. That's drawfast. The idea is, you know, it's a canvas. You can do stuff. You also have— I'm just going to refresh this just in case. You also have the ability to turn images into other images. So the canvas can be the place where that image is defined, and then it's sent into— come on. This is always the one that doesn't work. We're gonna try. You'll know it when it does work. And I'm gonna give it 10 more seconds and then I'm gonna give up. Oh, that's a shame. We'll come back to it. Maybe the models are asleep or something like that. No. Live coding. Kind of. Live something. All right, forget it. Ah, such a good demo too. Darn it. We'll have to come back. No, I'm going to insist. And it's going to say no. OK, whatever. The other one— damn it. Catch me after the show. So OK, you can draw it and it'll turn it into another image. That's cool, right? Good. Good cat. But what if you could turn it into other stuff, right? So probably the thing that went the most viral of these kind of AI demos that we did with, with, with TealDraw, and again, these are just demos like building on top of the, our actual IP, which is the canvas itself, was a project called Make Real. And the idea was you draw a user interface, you click a button, you get a user interface back. These models can write code and they can build websites based on specifications. But so far that was all like written, like make me a box, make me a toggle. My background as a, you know, product designer, like it's really hard to describe visual things in language the more complex they get. So with Make Real, what if you could just, you know, grab this stuff, copy it, go into another website. Let's see. No, not that one, not that one, not that one. One, go into another website, paste it, click a button, and have a language model, which, you know, we've told is an expert 12,000-year-old, you know, UI developer who's, you know, a bomb will go off if they don't produce a perfect, you know, UI, whatever, will return a working website based on whatever I wrote, right? So, cool. Little widget that was just generated live. Yep. At least that worked. Based on my drawing. You can do a lot with this. You can, you know, make little counters that, you know, I'm not even telling it how it works. It just kind of knows enough about how counters should work that it should actually produce a— like an operational It's kind of an intense counter, but okay, let's see how it goes. Just based on my drawing. Now, I could go further. I could say, you know what, actually, this is how the thing should work. I'm not going to show you what it should look like. I'm just going to show you how it should work. But you know what, actually, it fixed itself. I can add minutes and seconds and start it, and it should count down. Perfect. You might notice that there's a working website on the canvas as well. That's kind of cool, right? In TealDraw, one of the tech choices that we made, which is very unlike Figma or Miro or anything, and made my life much harder, was insisting that the canvas itself should be a website, should have like HTML and CSS. It's a React component and it's returning other React components. Every one of these shapes, you know, 1, 2, 3, 4, 5, those are all little React components, divs with SVGs in them, or in this case, an iframe. The cool thing about that is that not only can I put the website on the canvas and do all the normal canvas stuff, but I can also draw on top of it. So, I could say, um, no, no, no, make this serif for some reason. And— make an arrow and maybe make another arrow, lowercase please, and then maybe like a cat, angry cat apparently, and then feed this back to it as like the next iteration step. So it's kind of like, you know, designed by maybe by email or by post to say, here's what the website looks like. You know, take a screenshot and then send it back to me and let's see what it— yeah, right on. Let's see what you can come up with. Now the fun part about this is that I could just keep going. I could just keep iterating on this. And I've seen projects where people have taken this like 20 times, you know, like them and their kid are sitting down and like making a video game. Step by step by step by step. You do not have to be a— like, I haven't written any code. I can get the code if I want to. I can go like pop it up in like StackBlitz or something like that. And this is the actual code that it wrote on, wrote for me. But like, I'm not writing any code right now. I'm just drawing, annotating, you know, I can even, I don't know, like go to I want to go to Twitter. Go to Twitter, grab like a screenshot once it loads, right? Paste it in, just say like, yeah, you know what, just make that. Or whatever. And it will come up with something. It's pretty cool. I'll come back to that just because I see enough Twitter in my life anyway. Or like, you know, hey, make it look like Stripe or something like that. You can even use other kind of, like in this case, stop motion capture app, right? So it's not just little things. You can make fairly complex interactions. So if I grab all this and click here, and then while this is building, I'll show you exactly what's going on. What do I want to happen? First off, I wanted to use my camera, and I wanted to have like a feed of me on the screen. And then I want to have a button that I can use to capture, like, frames from that video feed. I want them displayed like thumbnails down here, and I want to be able to, like, you know, delete them if I want. And then I want a button that I can click to play. And when it plays, it should kind of, like, just animate through those different frames until I click stop, right? And when I'm in playing mode, hide the camera, do all that. So let's see if I have camera permissions on this tab. I do not, so I'm going to have to refresh the page. And then I will. Let's see if it works. There I am. So let's see. Boop. Boop. Right? OK, it's clipped, but other than that— Let's see. Terrible. But awesome. Like, hell yeah. Like, right on. I got time. I got a long slot, so we'll play this. I've run these things before the talk so that in case the models fail, we'll go back. Is this working yet while this loads? No. Such a bummer. Oh, hey, wait. Yes. Okay. We're working. Small intermission. Okay, cool, cool, cool. This is like slop slop, right? This is like turbo turbo. Okay. So the fun thing is that it's not just doing one image, it's doing— continually doing images. So here's a watercolor painting of a boring guy posing on a beach in Malaga, Spain. That's pretty good, right? But, you know, I can like— let's make them real big, real small, or do all the other things that you do on the canvas. Let's just make a lot of them, you know. This is— we had to shut this down because it was spending so much money. And it got really popular on like Russian Reddit, and I was like, I'm not paying for you guys, you guys can build this yourself. By the way, this is open source. If you want to— the SDK itself has a watermark in the corner, and you can pay us to get rid of the watermark, please do. But you could build with it without, you know, talking to me or anything else. And the extra code, the application layer code for something like DrawFast or MakeReal, that you could just read. I don't think of these as products. I think of them as demos. And they're also kind of like patterns. They're just a way of using the canvas that could be used for many, many different things. So yeah, that's DrawFast. I'm very, very glad that it did work eventually. Yeah, right on. Let's see, where we got— we got more. Oh yeah, Make Real. All right. I got 10 minutes left. I'm going to— oh, no, actually, no, I have more. As the last speaker, I kind of have to gauge attention and stuff like that. So if you're having fun, let me know. This time we asked for a user interface where we have a face just talking, and you can kind of type in your text. And we've given it some, like, complex, you know, like, here's how I want it to be laid out, here's how I want it to be drawn, you know, use these browser APIs to do text-to-speech. Here's exactly how I want the mouth to move, you know, like this way. When I first ran it without this extra figure 2, mouth scale, whatever. I'll leave this one out. Just for fun, I'm just going to do this one. You'll see how it does if you don't tell it what to do. But I love this progress of, you know, like, AIs are really, really unreliable. They will produce bad outputs all the time. But you can just add more data. In order to get them to do it right. And doing that visually is actually really easy, and it's the same way that you would communicate it to another person, to a developer if you're a designer, to another developer if you're a developer, I guess. So this is without the second figure. Sorry, zooming in and out can be distracting. Hello, hello. Hello, hello. What the hell? Hello, hello. That's not how you make a mouth move. OK. So then we did it with this. And let's see if— aweb. Hello, wee wee web. There you go. All right. So it figured out how to follow my directions there. You can do entire games. This is like a Missile Command type of thing where I'm like, the missiles are going to come down. They should look like this. There should be a little cannon on the— there, and it should have to defend itself. And let's just see how it does. By the way, this works with kind of any model. Right now I'm using Sonnet 3.5, which tends to be best with this type of thing, drawing, visualization, all that. Okay. So it might take a second for this one to come in. The other fun thing about, uh, that you can do with this is actually get the AI to use other AIs in order to do this. So in this case, I want a, uh, I want an ASCII art generator. Uh, I want a text-based picture, um, based on whatever I type. So I'm going to type like spider or something like that, and I want it to generate a picture of spider using text. Uh, there's no way to do that programmatically, but the models can do it. The, you know, you can ask ChatGPT to like draw you ASCII art. It'll be bad. It'll always be bad, but it will actually work. So that's what we're going to do. We're going to say user types in here, presses the send button, and then here I actually give it some code. I say, hey, talk to, talk to OpenAI, you know, use my license key. I give it my license key and I say, use this prompt. So I'm giving it a lot of instruction here in addition to the drawing, right? Really, the drawing is the simple part of this. But it works. And it's not like it's, it's different than if I just dumped all this text into a, into a field because it's also getting the screenshot. Things like priority or like all the types of visual notation, the visual language that you use to establish hierarchy or to, you know, like use Figure 1, Figure 2, Figure 3. That's a convention that is in its training data, and it understands that this is discrete from this, this is discrete from that. And as a result— oh, this looks terrible. Oh no. What the fuck? All right. There's no missiles. Where are the missiles? I'm going to try that again. I'm going to call that a bust. Boo. By the way, it also works with— I was mentioning the models. If you want, you can select different models from different providers. So we can compare OpenAI versus— that's looking better. OpenAI versus Anthropic versus whatever. Anyway, let's see if it did our ASCII art generator. I want a beach. You can tell I got the beach on my mind. Yeah, like, let's go. Two papers down the line, we'll be happy here. What did I say? Spider? But once again, like, I just ad hoc just made a little slop app from a drawing and from some text. And I have actually used this to make kind of ephemeral applications that I actually use. For example, I needed like a calculator for coffee grounds. Google it every single time. I'm like, I'm making 10 cups of coffee. It's going to be this much water. This is the roast. How many grams of coffee do I need for this water? And I just drew that up, clicked it. These are— every one of these things has like a permalink. So you can save this. If you do need an ASCII art generator, I think you can go to this URL, and that eventually will be the website where you could find that stuff. All right. We have more, though. So recently we started working on a project called Computer is the name. We've considered other names and we still might change this before we launch it, which is like workflows are all the rage and they, you know, they use a canvas to set up, I want this to happen, this to happen, this to happen, you know, and at the end, whatever, give me a picture of a cat. I think you could do a lot more with that. And as we played around with this, the fun part was that you could really basically create a natural language computer where it's— if you weren't technical, you wouldn't recognize it as a computer. It would just be like, oh, it's just doing what I'm telling it to do, right? If you are technical, you, like, within minutes you're like, oh my God, this is a freaking computer. I'll show you what I mean. So let's say, all right, here we go, right? This is a preview of this thing that we're gonna be launching. I have this complex flowchart, and I'm just gonna run it and we'll talk about it. What happens? Okay. We need a character name. I am going to say a lobster. Where does the story take place? Malaga? Trying to think of what a problem a lobster has. Problem needs solving. Lobster. Ah, duh. Find his lost pet snail. I don't know, man. Something visual, right? Okay, that just piped into this instruction, which is saying, hey, generate an outline of a children's story. Okay, hey, it's actually moving too fast here. Outline of a children's story based on the input. Write the first chapter. Okay, chapter 1, The Lost Snail. Here we're now at the write the next chapter using that first chapter as the input. The first chapter is also piping into this image. Okay, there it went. We now have the second chapter is being fed into another instruction to write the next chapter. Here's the image from the second chapter. Okay, we have chapter 3. We're all going to concatenate it all into text. I'm going to put a button there to let it— so take all that text, put it into one big text, and then we're going to take that text and we're going to do a couple of things with it. First off, we're going to generate spoken audio based on the input. Second, we're going to translate it simultaneously to Spanish, and then we're also going to generate an image based on it. So we have— here's our Spanish. We're going to take that and we're going to make audio based on the Spanish text and an image based on the Spanish text. So maybe the Spanish text image will be different than the English text image. I'm not sure why those didn't run. Let's see. Here, give me audio. Image. Oh, no, no. Try that again. Oh, come on. Demos. Demos. Fucking AIs. All right, but cool, right? I don't know. This is like neat. You can do a lot with this. This is actually really cool. Like, I don't know. We now have a stupid Spanish translation of a stupid Spanish story about a lost snail that was generated from my inputs, but through a pretty complex, like, piping of prompts and results through different things. Did this work? Sound of gentle waves lapping the shore. Bienvenidos a la playa de Málaga, donde comienza nuestra historia. Sound of a cheerful, slightly clumsy lobster. It's funny that the contextual stuff is all in English. Right, and then I can publish this and I can copy it and I can share it with other people. Here's a more computer-y example of this. Let's say we have an instruction that says, hey, take the current value and then do something to it, compute a new value based on whatever my inputs are. In this case I have +2, current value is 2, it should update that value to be 4. Wonderful. And I can do whatever normal computers do. Minus 1. This isn't very impressive, right? But the fun is that, like, it's not using code. It's not, like, using a deterministic way of finding the output. And when it's using language to do this, right? It's using its weird latent space of the models. And as a result, you can put non-deterministic strange things in and get to term, like, and still it'll still work, right? I can't in JavaScript be like, you know, 4 plus number of legs on a piano, right? Like that doesn't work as code, but it does work in this system. This should go to 7 because it knows how— blah! Number of legs on a piano is typically 0? Hold on, they have feet but not legs? Let me try that again. Number of legs on an octopus? Let's see if it gets that. Give me 8. Yes. All right, cool. We can also use other kind of abstract information as the inputs here. So let me turn on my camera again for this. For some reason, this really doesn't want to let me keep my preferences here. So once again, I'm going to use the camera as the input. I do not like seeing myself in these lights, but you get to see it, so I might as well. So now I'm just going to use the camera as the input, but I'm still going to compute a new value based on the current value. So let me try and do this. I don't know how to make a plus with my hand, but I can probably draw it on the screen. Plus. Let's try it. What the hell? There we go. There we go. Plus 2. Should get that as an image. Send it through this, like, procedure that's attached to this instruction. Yes! There you go. You know, you can do whatever you want. You can think of this as like, yeah, like roast me, right? Models will just give you information out and you can use that information as information in. If you've played with like noodle-based visual programming before in architecture apps, use it. Kind of Figma prototypes are like that if you think about it. Oh, jeez. Come on. That's not fair. All right. Whatever. What do you think I look like, huh? Yeah? Let's see what image out based on that. If it looks the same, then I'll be mortified. Ah, I'm better than that. Like, I've never actually liked noodle-based programming because you have these— it's supposed to be a simpler way of doing programming, but you still have all of the hard parts of programming are there. You have fixed inputs, you have fixed outputs. You have blocks that take certain inputs and produce certain outputs. You have types, you know, the text needs to go to the text thing, the numbers need to go to the number thing, things like that. We've designed this system to just take anything as the input and then anything as the output. So I can just keep, you know, adding stuff to this, you know, like if I wanted it to do a poem instead of, you know, roast me like a poem, like Shakespeare, whatever, I don't need to show you, but, you know, I could just put that in there and it would work with the inputs that it has. So it's like a really loose system, and it kind of like allows for this very sloppy ad hoc way of producing outputs, but like when it works, it works. The models are really good at this. I'll show you some of the other stuff that it can do. Just like my me picture, whatever this one, like you can use the canvas itself as the input to this. You know, in this case, I have a watercolor portrait, and it's pulling in my drawing as the input to that photo. So I can, you know, change this to be a— whatever, I guess a dolphin or something, right? And it'll pull that stuff in there as well. But again, you can use this as the input to anything. You could say— interpolate between these two drawings, right? Like, try and do my spider. All of this stuff is like places where the canvas itself, you know, I mean, again, it's my job, you know, these are its eyes. It's my job to sort of promote the canvas and show what the canvas can do here. And the— all of the features that we have, like, I don't know, frames or drawings or arrows and stuff like that. I want it to be more spider than dolphin. Let's see. Those are just opportunities to— come on. Might have to tell it what it is. Forget it. Dolphin spider. Forget my drawings. Drawings are confusing it. I want to know what 78% between a dolphin and a spider is. That's what I want to know. Let's see. Fair enough. Fair enough. Not a great response, but you got it. Cool. If you want to follow along with this story, because we're going to be shipping this within the next couple of weeks, and there is quite a lot that you can do with it, and it is both annoying and amazing. Models don't do what you want, and sometimes their outputs are terrible, but you can keep adding information to the system And that's an incredible power for, especially for people who aren't technical in the same way that Make.Reel allowed you to kind of like make and iterate on user interfaces just based on drawing, which everyone knows how to do a little bit. Yeah, this is actually pretty cool at building these type of workflows without requiring you to be a, to know what you're doing, basically, which is pretty cool. We are on X still, x.com/tealldraw, Bluesky if you're part of the new app. And yeah, I'm again @SteveRuizOK pretty much everywhere. We are based in London. If you are in London, let me know, or if you're coming through. And of course you can use TealDraw.com just like I am to, if you need a free whiteboard, fast collaboration, it's good, it's very good even. Please do use TealDraw.com. And if you are a developer or a product person and you have ideas about either integrating a whiteboard-like experience into your app or building something new that just doesn't involve building all the undifferentiated stuff that's normally involved in the task, tealdraw.dev is the place to go, and we have a really good Discord as well. And I, oh yeah, I shared the slides, so there is a link to these slides up on, on TealDraw itself actually, so I think, yeah, you guys can probably, I don't know where you are, but you could probably see my My cursor. It should be a read-only link, fingers crossed. Oh no, it's not. All right, we're out of here. Show's over. Right on. Well, let's see how many people we can get in here before it breaks. And I will leave you with that as a thing to play with. All right. Thank you. See ya.