Agents on the canvas in tldraw with Steve Ruiz

When I met Steve yesterday, I was interested to hear more. It's great that there is a company doing something that's very cool, TL;DRAW. Steve Ruiz. Thank you. I'm the last talk of the day, which puts me in a kind of an interesting position. We've seen a lot of talks, maybe flagging, whatever. I'm going to need you to kind of let me know whether I should keep going. If everyone's yawning, we're gonna go, we're just gonna go, we'll just cut it. So please do give feedback. I am Steve Ruiz. My talk is Agents on the Canvas. Some of you probably follow me on Twitter and so you've seen some of the stuff that we do at TealDrop, but I'll give you kind of a really quick intro and then we'll jump into demos. So this is TealDraw. TealDraw is a website that you can go to. It's a free collaborative whiteboard. You can share links and use it. It should be pretty good as well, which canvas apps really, really, really does matter. So you can go to tealdraw.com. You can use it. But that's not really our product that we commercialize. We actually commercialize the tealdraw.dev, the SDK. So you can use TealDraw's canvas and build other things. With it, so we license it to other companies. Everything from ClickUp building their whiteboards feature all the way to crazy stuff like Grant Cott making weird liquid simulations that use the whiteboard geometry to actually play around with the canvas. I can grab this and kind of— there you go. You get the idea. Very fun. Kids love it. We make an SDK and our value proposition is, hey, you can use this and build cool things with it, which puts me in the lovely position of having to prove that, that you can build cool things with it. So we do a lot of demos and a lot of those demos are AI related. You might have seen Make Real. This came out in like 2023, right when the vision models dropped. And the basic idea was that what we could do with with tools like this is grab the shapes off the canvas and copy them as like a PNG and then go into something like Cursor and say like, hey, can you look at this image, like build this. And it would do it and you'd eventually, you'd have your little login screen. But that's no fun. What if you could do it in the canvas as well? And so this is using Sonnet 4.5. Let me switch down. I'll leak my keys here and roll them later. I'll grab a faster model so that it finishes for sure. And the cool thing about TealDraw's canvas, by the way, is that it's just a React app. Like everything, like that's a React component. That's a React. They're all React components that just return SVGs. Or in this case, they return like iframes. So you can put— and it's really on the canvas. It's there. I can draw on top of it, which we'll see in a minute. And so even with my little welcome back here, in a second we'll have a little working piece of website, which actually is pretty good, better than it was when I started. What's fun is that I can— oh, god. I can also do things like— like this, which you can't do in Cursor. Although we should talk because more and more apps are starting to do that. And then you can use that as the next prompt. So like, you know, what we saw were people putting like screenshots of like Stripe, you know, be like, just make it look like this, you know, and it'll do it. It's really great 'cause you're just building the prompt, you know, it's still gonna send it as the, PNG, right, the screenshot. But it's also going to send the HTML and the CSS and stuff like that. And so it'll know to modify its previous work. Actually, we have Flash doing it. So you can see Flash isn't quite as good at it. So you'll have to take my word that there's no more welcome back there. So that was one of the first things that we did, but it wasn't the last. The next big thing that we were— there's a couple of little things in between. But the next big one was TealDraw Computer. How many people know about TealDraw Computer or have seen this before? This is awesome. This is why I do the talks, 'cause that's like nobody. And TealDraw Computer is fucking awesome. Like you gotta play with this thing. You gotta show it to people. So the basic idea is that you have a canvas and you can have things on that canvas and they can connect to other things via arrows. And then that can kind of form a, kind of like an executable flowchart of different stuff. So if I start running this thing, it's gonna take that terrible picture of me, it's gonna take this little drawing of a cat that I made, it's gonna create an image, hooray, but then it's also gonna create a 3D model based on that image. It's also gonna go into this guy, which just says write a poem, and then it's gonna write that poem. So each one of these blocks has a procedure, kind of like a based on your inputs, do this. And then that procedure can be executed, which is what we're doing now in the context of its inputs. In this case, our image block has a built-in procedure that says craft a Midjourney prompt or something like that. And then it generates that. But this instruction is interesting because that came up with its procedure based on what I wrote. And so now a man with a beard, playful grin, tongue out, goddammit. Right. No words are said, a silent jest. Okay, cool. And you can keep going with this, right? And you can even have cycles and you can even have conditions, right? 'Cause we have like, you know, or I forget, screen is too zoomed in to see it. You'll see it in a second. And so even though we kind of started with this idea of like just workflows, it quickly became like, oh my, you could just do computer stuff with this. It's just that there's no code. It's just we're using an LLM for everything. You know, you can make the world's most expensive counter just by having state and having a thing that loops through like that. I really wanna let this thing finish, but I'll show you something else. This is a Frontier math solver using, GPT or a Flash, Gemini Flash. So logic puzzle, kind of complex, but it's going to go into this block where it's like, hey, you're supposed to complete this task. Look at the state. Ignore the cat. Look at the state. Is it done? Like, is the task complete? And if the answer is no, then, hey, you're supposed to complete this task. This is your input. Take it, right? And then I want to count the turns, so I'll update the counter as well. And then when it's done, it kind of spits it out over there. Oh, it's going to be so close. Oh, well, it's going to be terrible anyway. It's just like a Three.js thing. You'll have to take my word for it. And this will also kind of take a minute to run, so I'll come back to it. But you'll notice that we have these conditional blocks here, right? Yes, no. We also have a third state for that, which is maybe, because again, —like, there's no code here. I mean, there's a lot of code here, but there's no— like, the execution is being done by a language model. So it can kind of be in this kind of 6/7 mode of both true and false at the same time, or maybe, you know? So it's not done. It's going to try and solve it. Then it's going to go here, and then it's going to loop back around to here unless, of course, it is— done, in which case we're good. So that is TealDraw Computer. You could use this to make all sorts of really great stuff. It's free. Just burn my Gemini Flash credits or whatever. I think someone asked me like, hey, how long is this going to be free? I'm like, I don't know. It's been running for a couple of months now, and I've spent like $400 on this. And like, Google gave me like $250,000 worth of credit. So you got it. All right. So, oh, I guess it did one-shot it, which makes it a bad demo. But anyway. The other thing that we wanted from the canvas is we wanted a collaborator, essentially. I did have slides, but we'll kind of blow through them. That was computer. We're going to fairy draw. Trust me, that'll make sense in a minute. So, we wanted a virtual collaborator, essentially, right? That's the reason why, you know, canvases are so popular is because they're great for collaboration. They're great for collaboration with people. Why shouldn't they be great for collaboration with AI as well? Well, the models are not trained very well on visual stuff. There's just less of it. And it's also really confusing. You know, even like left and right is a really confusing idea for models because there's like my left, there's your left, there's like stage left, there's, all sorts. Even like the idea that in the web, like the y-axis goes up as it goes down. Do you know what I mean? Like 0, 1, 2, 3, 4, 5, 6, right? That's backwards for the models. The models are used to Cartesian graphs where the y goes up in this direction, right? And negative down in this direction. So coordinate systems, all that stuff is really hard for it. Yet you can kind of bully it into being kind of good at it. So this is a— This is an old model's generation. I'll clear this stuff out. What's fun about building AI tools on the canvas is that the models keep getting better and I can just keep the same UI and just update the model and then it just gets better underneath you, right? I want this to finish my storyboards. So this was the first one, first one that we did. And this actually was really, really hard to even figure out how would you do this because you can send it a, a screenshot, but then it has to send something back. And also, the screenshot's not really enough. So we have to send it a textual representation of the canvas. But the actual JSON for the canvas is too complex. So we have to send it a simplified version of that. And yeah, there he is in his driver's seat of the— POV through the windshield. Not bad, not bad, not bad. And it's kind of like the pelican thing riding the bicycle. Do you know what I'm talking about? It's like a good— but a whole app based on that. It's not good at that. It's not good at this. But the cool thing about this is that it's creating the same things that I can create, right? It's creating boxes and— I guess I haven't showed it off, but like, you know, cool like ink and things like that. I don't know what happened to its body, but I'll give it a body. You know what? We'll work together here. And then the demo that I always do is like something like this, where— and then I ask it very nicely, make the cat blow out the candle. And I didn't tell it that it was a candle, but it should be able to see, and it should be able to figure out what that should look like. And we will see. So anyway, we had to come up with this really complex— Figma can't do that. Figma can't do that. I can do that. Figma. It's a joke. It's cutting edge, believe it or not. But yeah, there's a whole textual representation of that. Anyway, this is what we did before. And then we decided to do something a little bit more recognizable. And so we created a cursor agent. So the difference between— I'll let everyone see this because I'm sure you've never seen this before. This is a sidebar, an Agentik agent sidebar where you can ask questions for it. Make me a state chart showing a cat purchasing and delivery app. So this was like the next turn of that same technology of saying like we need a way of representing the canvas content in text. We need a way, of teaching the language model to create that information, but also to consume that information to be able to understand. All right, so it's still trying to fit it into my viewport. You could see that, but I was zoomed in, so it's really tight. So I can— let me try and spread this out a little bit. All right, that looks— you know what? That's not bad. That qualifies as a state chart. You can see it reviewing its work. Also, right? And making the diffs and all that stuff. So it might actually be confused by the fact that I moved things, but it has a to-do list, et cetera. And hey, that's my cat purchase and delivery app. Like ready to go. [APPLAUSE] However, this is still not really like a virtual collaborator type thing, right? So we went from just the prompt, then we had the agent. Now I want the agent to be on the canvas. And when we first thought of this, you know, we were like, yeah, it'll just be like, just like a, you know, just like one of us, right? Like if I join the same session in a different tab, like, you know, like we support multiplayer really well actually. But that's not what, that's not a human. That's not even close. It's much more like a fairy. English fairy, not one of those scary Finnish fairies. Slightly mischievous, right? It doesn't work for you, but it kind of likes you. You can ask it to do things, maybe it'll do it, maybe it won't, maybe it'll do something else. So that's the modality that we've gone with for now. Someday maybe there'll be, people, but they're not, they're not nearly. So you know what? I got a ferry right here actually. I can kind of just pull it in. It's way over here. So let me grab that, kind of come over to where we're at here. I can do the same thing, you know, like draw me a cat and it'll think about it. Do you see it? Thinking, which is actually not a trivial thing. I mean, I don't know if you've ever run 5 Claude at once or whatever. I mean, we saw multiple agents before, but actually some of the same problem or the biggest problems become your own attention span, your own memory, right? You become the bottleneck of which agent did I ask to do what and which one has context on what, et cetera. So the idea of having these agents be, and like which ones are running, which are done, all those types of things. So the idea of having an agent state be kind of visible and scannable and for you to be able to, let me see if it can do the candle thing or, ooh, you know what? I'm gonna give it something else. This is new. Never done this one before. Make it spill the water. I'm curious how it thinks about fluid. Yeah, is it thinking? Is it waiting? Is it flapping? What's it doing? Your cat is now making a mess, knocking over the water bowl. Classic cat behavior. That's a real problem if you want to run these things, and I want to run I want to run like a bunch of these things, right? Okay. But we'll get to that in a second. So I think it's interesting to think about like how does this work, right? So we have the normal agentic loop for the ferry, individual ferry, where you send it a prompt. We need to, while we're still on the client, right? Because here we are on the client. Kind of decorate that prompt in a way because we're gonna be doing the, creating the instruction set, you know, and then executing it on the clients. But there's quite a lot of information that we can send to the AI so that it can make that decision. So user's time and date, which other ferries exist and where they are, the recent things that the user's done, the chat history for a particular ferry, where the user's bounds are, but also where the ferry's bounds are. And I'll get to that in a second. Screenshot of the viewport, the canvas content, but in a way that is like with diminishing detail as you get further out of the viewport, which I'll get to. And then, well, I'll come back to this, but the shared to-do list. Because a big part of collaboration is collaboration not just with me via prompts, but with each other. I want multiple agents working together, right? Which is very hard to do, by the way. This kind of coordination doesn't really work very well in documents. And so we haven't really seen it very often. So again, cutting-edge stuff happening at TealDraw. Also abilities, what kind of fairy wand does it have and its personality? So for example, this personality is friendly and helpful. And it has ears. But we could give it an antenna. I don't know if you can see it over there. Because you want to be able to tell the difference between these guys. Pointy. All right, whatever. I'm going to go back to antenna. I like that. So anyway, we have to give it all the information. Then we have to send it to the AI. The AI builds that system prompt based on what it has. Actually edits its own message history in order to kind of like compress all the stuff. You don't really need everything. Generate those actions and then send it back. The actions that it can have are simple things like creating, updating, deleting shapes. We've also given it a lot of like positional stuff that helps those flaws that I mentioned before about like coordinate systems. So for example, rather than having— if it wants to make something like underneath this, rather than trying to figure out all the math itself, it's kind of bad at that. So we give it like a put beneath, put this thing relative to some other thing using this amount of gap or something like that. And it's much better at that. So there's quite a lot. And of course, because it's a loop, it can kind of schedule a review or it can schedule a next action or it can send a message to a different agent or a different Very, and then we handle those actions on the client. And if there's another request, for example, for a review, then we start it all over again. That's the basic loop. The only real difference is that there's a lot more information that goes in, or maybe not, I don't know. There's a lot of information that goes in just because it's a 2D document, right? I mentioned the canvas content is kind of, kind of complicated. In a canvas like this, there's more information than the ferry kind of needs. And so we've given it kind of like this ability of, OK, I know where my viewport is as the ferry. Now let's reduce the information that we're going to send based on what's in the viewport, what's outside of the viewport. And so we kind of cluster shapes into— clusters, things that are overlapping nearby, the proximity where the overlap gets— in order to be a cluster, increases as you get further away from the viewport. And so we're essentially telling the viewport less information or the ferry less information about those shapes that are further away. We're saying, hey, there's a bunch of shapes over here. There's 9 shapes over here. These are its bounds of this cluster. There's like 12 shapes down here. These are its bounds. But the things inside of the viewport, we send it what's called blurry information, which it turns out it's fine. You just tell it where it is and the type of thing and then the text if it has a label or something. Because we also are going to send it the screenshot of this. And the model is really great at kind of connecting. Like, OK, this box over here is blue. This is the ID that it told me about, and I know its bounds and all that. And so we only share most of the information when the shape is actually focused. But even then, we're trying to reduce the amount of information that we send. And this is how you get— this is how you get fairies drawing on the canvas, right? This is marked as what's next, but I'm running a thing that was just landed like 20 minutes ago. So I actually get to demo the multi-fairy collaboration. But yeah, basic idea is that one fairy good, three fairies better. You want to— again, like the— it becomes StarCraft, right? Like I want as many agents as possible running maximally as much as possible, right? Until I become the bottleneck and just start to freak out. And I also want this to work with other people, which it does, which I'm not gonna be able to demo, but every collaborator brings their fairies with them. And the result of that is chaos, right? But the thing is, it works. You know, we did a retro like 2 weeks ago and it was like 15 people, 15 fairies, 1 fairy per person. We were in the same document and it just worked. It was silly, it was fun, it was crazy, but I never had a collaborative experience. It was a real computer moment, you know what I mean? Like 15 AIs collaborating with me and 14 other people all at the same time. And it just worked transparently because the canvas is good at that. So I think the things that we are looking at a lot for this places where these types of problems of running multiple agents, for example, have been solved before. A lot of real-time games, a lot of like CRPGs like Baldur's Gate where you have tons of entities happening in real time, you need to know the state of those entities. You need to know like be able to differentiate between them, select the one that's good at this or they're good at that all in real time. And so we're looking hard at accessories kind of dynamic animation systems in order to make sure that there's enough difference for you to kind of know your— differentiate your little Canvas agent. Also, not all agents are the same. You want to have certain ones with more context on certain tasks. And so we— yeah, it just sort of becomes much more like a game where you're kind of getting your squad together, right? And like, oh, man. Which one of my fairies is really good at diagramming? And everything else kind of maps onto as well. So even, you know, like, okay, how do we represent, you know, connecting to like an MCP server or something like that? Am I warging into Notion or something? Am I sending a little butterfly out to get that external information? Subagents, of course. There's a difference between which are the agents that I'm gonna communicate with directly versus which are the agents that are only going to be kind of managed by that first level agent. And also things that are unique to the canvas, right? I want to have a certain area of the canvas where there's a kind of an agent attached to that or managing that, right? If anything enters this zone, I want this fairy to kind of like do something about that. And then that's kind of how you get to a potential fairy computer, but we're not there. But you can imagine that user prompt leads to something over here, which has this kind of like agent-based automation. Anything in here, make sure that you translate to French, move it over here, anything here, export it as a PNG, put it in this folder, et cetera, like that. So they're composable fairies. They're not just individual. They're primitives that you can use. Also the idea of just putting context somewhere and saying, hey, if you're working in this area, like read this, like leaving a note, leaving a message on the canvas for the agent working in that area to pick up and, you know, kind of use if they're working in that area. I'm gonna do a demo demo, but just in case I forget or run outta time, TealDraw.com is the website where you can use this. We are gonna try and somehow put this into the app, possibly just for the holidays, but you should be able to play with it by the end of the year. And then TealDraw.dev if you wanna build with it, which is how I get paid, or how we make our money. I get paid either way, it's great. Go to TealDraw.dev, build with stuff, it's really, really good. Lots of people are using it. Yeah, let's try and do something here. So I'm going to ask one of these fairies to— I'm going to reset this chat here. Write a PRD for a cat— what was it? Cat sale purchase and delivery app. Kind of like a CADO for cats or something like that. All right, so it's thinking about it. Where are you? You're here? So that's pretty far away from every— no, it's just beneath. OK. Where'd it go? Oh, it went back to where I was. OK, right on. So you can see it. Sorry for everyone over there. I'll kind of run out of the way here. It's thinking. It's going to work. Its name is Gumpspark Snellgold. Asking, okay, so it's asking the other fairies for help. I didn't expect it to do that, but I, that's new. So it's, I think it's, you know what, it did, it kind of changed itself. We don't represent this, but it kind of like an ant colony, oh God. Like an ant colony, you know, like if there's not enough soldier ants, like an ant will just become the soldier ant. And so here we got the user journey, someone's doing that. Someone is doing goals and objectives with personas, which is great. Michael the breeder. All right, so we have a lot of information here, a lot of information, a lot of, a lot of agentic stuff going on. In fact, we can look at the to-do list that it made, and it's requested help from all these other ones. Glimmer Gum Bolt Spark is working on the delivery and logistics section. Flick Flick Spark Bark is doing the technical requirements. So we have a lot of information. It's made the mistake of doing it all on top of each other, but Luckily, they don't— it doesn't matter to them. They'll read it anyway because we give them the text. So we're getting pretty close here to the next step, which will be doing the wireframes. Good. Good. Awesome. That's what we want. Perfect. This is a— I mean, I hate to say it, but this is the future of agentic workflow. So get used to it. I look forward to FaerieCon 2026. But you know what? What's cool is that they are recognizing that things are overflowing, and so they're kind of cleaning it up. They're moving things out of the way. And I think we're almost all done with everything. By the way, I'm about at time. Should I keep going on this? Yeah? Okay. Yeah. All right. We got it. I'll do that. Let me, this time I'm gonna be more explicit with it. Hang on, get out of there, get out of there. Get out of there. Are she done? It's done, it's done, it's done, it's done, it's done. All right, I gotta tell them to stop. I don't wanna remove it, I just wanna, Say, be done. This one's done. All right, we're starting to finish up here. I'll just ask these three 'cause they're done. I need this. Yeah, you can see the log. It was awakened by another fairy. It worked for 141 seconds. Now it's reviewing its own work. So the whole agentic loop that we showed before is still working here. I'm just going to ask these three. I'm going to nominate a leader, which is going to be this guy, Glimmergun Boltspark, friendly and helpful. I want you guys— hang on. I got to promise myself I'll use this more. I want you guys to do the wireframes using the boxes above. Feel free to add new wireframes, but just make sure that it reflects the PRD elsewhere on the canvas. Okay. These guys are done. That's not bad. If your project manager sends you something that looks like this, you know that they made it in TealDraw. So they are— how are these guys doing? They're thinking about this. They found the PRD. They're going to review that. It's probably going to take them a while to gather all the context they need. They're breaking down the wireframing project. This one's the fairy orchestrator, the leader fairy. So it'll just start tapping in the other fairies in a moment. Just created a comprehensive to-do list. So they've added items to the to-do list, and now they need to start the project. Flickr Tinker, Fuzznose, and Tooth Toe Windbubble are working on it. I think they should have each a different wireframe to go for it, but here we go. Here we go. Oh, that's pretty good. That's a definite form for checkout. So as this completes, just to wrap it up a little bit, TealDraw is a very good whiteboard. It's also an excellent SDK if you're building anything like this. Some of the best companies in the world are building products on this. The canvas, very fun. Also, like, people like it. And you can do real work on it. And, you know, it's never been a better time to build. And you should give it a shot. TealDraw.dev. And also, this is 4 agents before we had like 7, like working on the same document and actually kind of like being able to see what each other are doing and coordinate using a shared to-do list and all that. So whimsical, definitely whimsical. But these are real problems that are really hard to do in other kind of paradigms. So hopefully things continue to improve and maybe you'll see some more Canvas stuff in the future. I'm Steve Ruiz. Catch me later. Bye.