Steve Ruiz: The TLDR of the Collaborative Whiteboard tldraw

Hey everyone, and welcome to Generative Now. This is the show where we talk to the builders who are creating the world's most exciting AI products and companies. I am Michael McDonough. I'm a partner at Lightspeed, and for today's episode, I talked to Steve Ruiz, founder of TLDraw, an open-source whiteboarding tool that has exploded in popularity among the developer community over the past couple of years. In this conversation, I learned how Steve's background in fine arts led him into the world of product design and prototyping before iterating his way to TL;DRAW. We also talked a lot about how TL;DRAW has even now transformed into a new type of prototyping tool enabled by GPT-4 Vision. Take a listen.

Hey Steve. Hello.

I have so much to ask you about TL;DRAW. Wanna learn the whole story, wanna talk about Make.Real, kind of the, the whole thing. But I, I would love it if you went all the way back. Uh, tell us about the early days.

I'll do my best. Okay. Um, cool. Yeah. So my name's Steve Ruiz, uh, founder of, of TealDraw now. Um, before this and kind of before tech, uh, I'm from Chicago. My background is actually in fine art. So I did my undergrad in painting and drawing. I did my graduate studies, my MFA in visual art, painting and drawing at the University of Chicago. I also did a lot of writing around art. So I would go to art shows and write reviews for magazines. And I organized like a calendar of all the events that were happening in Chicago and stuff, which is still running 14 years later. Wow. That's pretty cool. WordPress, man. Yeah, it just never goes away. And I also, I used to build like artist websites for them. Nothing special, but I had like just enough, you know, technical skill to be able to do that. But, but barely any. And yeah, so I was doing that in Chicago until 2015 when I moved to the UK. I moved here because I met a girl who ended up being my wife and we settled out here. She was working at Cambridge. So I was living in Cambridge and had a nice little studio there. And that went on until I was around 30. I decided, okay, well, you know, painting, drawing, like, love it, but it's, you know, it's only kind of like one part of what I like to do. There was this other kind of more analytical, maybe technical part, and also more social part, which I really wanted to exercise. And painting and drawing, you know, as much as I loved my studio, it was just me. So I kind of made a pivot, closed the studio, started getting into more design and did a kind of like, just not a crash course, but just kind of like just smashed my way through, caught up with like 10 years of essentially design since I'd last really thought about it. I ended up getting into this kind of interesting territory of like prototyping. At the time, I think Uber had just started using like Framer, like the old, old Framer that had just spun out of Facebook, um, in order to like prototype these really complex interactions around the first versions of the app. And this was clearly like the hotness at, at the moment, which was like late 2016, early 2017. And I was, I was, you know, seeing meetups around this, around the world, people talking about it. But no one knew how to use this, this application. No one knew, no one knew how to make this stuff. Framer. Framer. Yeah. So it was like a CoffeeScript-based little development environment. But it was wonderful. It was like a little picture of an iPhone on the right and then code on the left and you know, it was doing hot reloads, all that stuff that hadn't been around the last time that I had started doing any programming stuff. So that's how I basically, that's how I learned to program. And I just got really hooked on it, got really into this space of like prototyping interactions and essentially like doing that design work that like necessitated a certain amount of building in order to really make those design decisions. Um, it also turned out as I worked on this more and as I talked to more people that a lot of the art training that I had, not necessarily the technical, like painting and drawing, but, but more in the, more in the conceptual and performance side of things where you're really thinking about how someone is experiencing something. Um, uh, all of that actually came in like really, really handy and, um, kind of became a strength that I brought into, into every design project. So, uh, that was around 2017. I worked for a very, very strange experimental film company that was building like a digital component of their total theater experiment. I don't know, it was very strange. What was that? That was, uh, Phenomen Films in— they were like a Russian-Ukrainian project making, um, a film project called DAU, D-A-U. Which was, um, you can look it up. It's extremely strange, experimental. They spun out like 10 films from like 6 months of recording people living in a scientific compound that they built in Ukraine. Extremely weird project, extremely arty. For me, it was a great bridge between that part of my life and kind of like the next part of my life because it certainly was, uh, full of these really interesting technical design problems as well. And then around 2019, right before my, uh, my daughter was born, I, uh, actually ended up taking a job at Framer, like the company that had kind of made the software that I'd become an expert on. And that role was a, um, kind of an education role somewhere between like design relations, developer relations, and product education. But what it ended up being was a lot of, um, trying to make content, like educational digestible content for Framer about that product, but also recording a lot of tutorials and videos and learning how to speak in front of a microphone and learning how to edit videos. And boy, that's tough. But it did get me thinking a lot about how you tell the story of a product and how you kind of like communicate that from like an education product and marketing perspective. Which ended up being very useful later on. So, uh, 2020, I worked for a company named Play, still in the design tool space. It was a design tool for your phone. It still is, but it also is on desktop, uh, with the premise being that you did the design, that you did the prototyping, everything with like actual native, uh, elements, um, on your phone so that the, the fidelity between like what you what you designed and what you ended up with, this prototype, was incredibly high. And that was great. It was a wonderful design team, really, really challenging product, but in ways that, I don't know, there are elements of design tools that really haven't been rethought since really early on because it's all desktop. You have a layers list and all that objects list. But on the phone, we had to rethink every single one of those design elements, which was really great. Highly recommend that you check that out, by the way, just to, to, to see what a design tool looks like on your phone. Um, while I was working there, I was contracting. I was also doing some open source on the side and the type of open source that I was doing ended up drifting more towards the, the, the visual side. Um, not necessarily in terms of like graphics, but more of like little tiny problems that, that really hadn't been solved yet in a good way. Um, one example was arrows. It ended up being fantastically complex and super, super fun to work on. And I still had all that, um, content creation, uh, muscle, like really, really well worked out from, from being at Framer and being at Play. Uh, and so I was like just tweeting about this and sharing this journey as I was going and people got really into it and they're like, hey, arrows, never thought about 'em, but turns out they make, uh, fantastic, like snackable technical content that within 8 seconds or like a couple of GIFs could really communicate, or at least make you feel the depth of a problem in a way that, even if you don't necessarily care about arrows, like it's still nice to have that feeling of getting it or thinking into someone else's problem for a little while. So that ended up being an open source library called Perfect Arrows, which is still used. People use it and people like it. And it's just a nice little, example of, like, I don't know, a small visual problem solved in a— as best as I could.

Can I ask clarifying question before you move on? So, you know, you mentioned early on about how you weren't that strong technically. Obviously, you came from a design background. At this point, like, are you building this stuff yourself? Oh, yeah, yeah, yeah. Did you learn how to do that through this journey into prototyping and learning all these prototyping tools?

Is that how you made that leap? I tend to pick things up pretty fast and I tend to work on things kind of obsessively until I've picked them up pretty fast.

Like a lot of entrepreneurs.

You just have to. I mean, now I'm doing it with sales or something. I don't know. Yeah. So especially working at Framer and being in the education space, I just had to learn stuff well enough to talk about it and to teach it. And, you know, as it's I wouldn't be the first person to say it, but like you learn a lot just by having to put yourself in a position where you're teaching it to others. And yeah, I would say by now I was kind of at the level of like a mid, mid-level. I could have walked into any, any mid-level like web development job. Got it. But I hadn't really worked with this type of like these little visual problems, but I got really into it. Like I got really, really into it. you know, how do you draw a line? How do you, how do you, you know, these, these vectors, these like directions and stuff, like linear algebra. Um, obviously with two art degrees, I never really got spent too much time working with math, but I kind of had to learn it. I think practically, like I knew what I wanted to see. I just didn't know how to do it. And, uh, and then I just had to figure out a way to find those answers. These days I would ask ChatGPT.

Yeah, me too. You know, when you really went deep on Framer, and these prototyping tools, was that, was that in an effort? I think you mentioned it was, you know, you sort of like going back into interest around design and things like that, but was part of it so that you could act on these ideas to build products before you had the knowledge to code? Or was it really just to kind of learn about these tools specifically? Does that make sense?

I mean, I did the whole career switch thing. I kind of surveyed the landscape. And came up with a strategy. And that seemed to be the place where I could go the furthest. And that seemed most interesting to me. And that touched on so many other things. So it was to learn product design.

It wasn't to go and like build products and start a company.

No, no, no, no, absolutely not. This was— this is— got it. Specifically, like I had identified this problem, which was that development was expensive. that design could only go so far because it was producing like static assets, you know, images. Uh, and that there was a space in between those two things, especially around more ambitious, uh, interactive types of applications where, you know, if you could have a design-minded person working in the code to sort of, you know, make those decisions about, okay, here's how this stretching thing should feel, or here's how this animation should play out, or here's how this interaction should work. If you could have someone within the space of design making those decisions by building it, then you would have a better answer for saying like, okay, now engineering, like, you know, take this and productize it. And that type of lightweight, lightweight design coding was, was really attractive to me. And again, everyone was into it. No one could do it. So it seemed like a clear strategic move for me to insert myself there. Yes. And it turned out that I really, really loved it. But these sorts of like, and still today, like this idea of prototyping, of building something that was, you know, we don't really know what this thing should be. So we're going to build it in order to find out what it should be. And if we, if it turns out that we need to throw this out and do it again, then we will. If it turns out that what we built we can continue to build on, then that's great. But it's this constant, like, churning approach to development that actually works pretty well for products as well. Right, because you can get real feedback, right?

Like, not conceptual feedback, not hypothetical feedback. You can put things into hands, like, a lot quicker and a lot easier. Yeah.

And even the attitude that you have while building something that you think is a toy, Uh, tends to be more creative and more, like, you have a little bit more willingness to push a little further on something that you think, eh, I'm just going to throw this out anyway, you know, and start over once I, and, uh, even if, even if the thing ends up being the product and we, you know, it ends up shipping and stuff, uh, like, you know, Tildraw.com, I still largely consider like a, a demo, you know, but it's like a demo that has like 6,000 people using it a day, uh, which, you know, it's not a huge amount, but it's also not nothing. But it, it kind of gives me the permission to, to experiment a little bit more with that. Um, right, right. Yeah.

Cool. Well, thanks for taking, taking us on that detour. So Arrows. All right.

So I'll, I'll, I'll accelerate. There's not, there's not actually too far between Arrows and where we are today. Okay. Uh, After Eros, I started working on a telestrator. I don't know if you know the word telestrator. I do not. Okay. If you've ever watched American football or especially American football where they, they do a replay, they like, they draw on top of the screen. Exactly. Precisely. It was invented by a guy in the '60s and he called it the telestrator. And now that's the name of, of that tech that, that is able to do that. It's a lot like drawing on your screen when you're sharing your video on Slack or, you know, I think Zoom might allow you to do this these days. But I was recording, I was recording some streams and I needed a way to draw on top of my screen live in a way that was not attached to any particular application, but just like everywhere. And I couldn't find a good good result for this. So I, you know, I had code lying around that I'd built for like a little drawing app. I knew that you could make like full-screen Electron apps that had like a transparent background. And so I'm like, all right, I'll just do that. And I made something that I called Telestrator. Couldn't think of a better name. And it, yeah, it still works. You can still download it. It just lets you draw on top of your screen and then switch back to whatever you're doing. I also bought a domain for that. I thought I was going to call it like Teledraw or something. And I couldn't get that domain. Couldn't find like the right, you know, collection of, of things. But I did find TealDraw. So I was like, all right, I'll just do Teledraw and call it TealDraw. So I bought that domain. I never used it. But part of that was that I was drawing on top of my screen and I was using like a stylus, like a drawing stylus. To do it, like one of those like Wacom pads. And I knew that the Wacom stylus that could do pressure, kind of like the Apple Pencil, you know, if I pushed hard in Photoshop, it would get big. If I pushed lightly, it would be small. And I was like, well, I want, I want my little Illustrator drawing app to, to work the same way. So I started looking at first off, how does that work? Is there any open source kind of packages, libraries that I could use in order to, to solve that problem of like saying, well, I have all these points that are where I know my mouse was. I have pressure for each one of those points. You know, give me a line that gets thicker and thinner as it goes. Uh, turned out it was an unsolved problem, uh, at least in, in open source world. And it was pretty poorly solved everywhere. So I started working on that again, kind of in public, again on Twitter. Sharing like the terrible results at first and also sharing all the little like, oh, I've just figured this out or this out or this out. Hmm. When is this? Fall of 2020, I think, is when I was working on this.

A little more than 3 years ago.

Yeah, yeah, yeah. Gosh. And of course, it attracted a whole bunch of interest from people who actually care about this stuff, who are able to, you know, do the geometry and read the math and tell me, okay, no, this is actually how this worked, or I wrote a paper about this 10 years ago, and here's my Flash game that I did that explored the same topic and all that. And so I just was incorporating all this advice and kind of tutelage that was coming around from other more domain-specific experts, or domain experts, and ended up with something that was like, it was really good. And it was really good in part because like we figured out the math, but also because like my, um, I just worked with ink for, you know, 10,000 hours as an artist. And I had a really good feeling for, ah, this is how it should work, or this is how it shouldn't work, or this is like, this is what good looks like. So I called that Perfect Freehand. Um, and now that's used everywhere, but even again, like this, this joke of saying like, it's perfect, it's solved, you you know, this thing which is highly subjective and highly like opinionated. And there's, of course, there's no like perfect looking digital ink or there's no perfect looking arrow or anything, but it's, uh, it's just kind of my, my winking way of saying like, um, that this highly subjective aesthetic thing is, uh, is as good as I could make it. And, um, and like, you know, you can do all sorts of stuff with it. If you wanna play around with this, this is at perfectfreehand.com. Is the demo. Um, but yeah, it's used in like, uh, draw.io uses it, uh, ClickUp uses it, uh, uh, Canva uses it. Wow. Um, I think Vercel was using it for a while. Like, um, it's just significantly better than, than the next best thing in this, in this space. And there really wasn't much of a next best thing in the, in the first place.

And, and so at this point you're really, you're really just setting out to, to do, to be able to do this form of drawing on a canvas, on, on the web? Is that the idea? Like, what, like, what's the goal at this point? Or is, or is there no goal at this point?

What I was starting to find out was more related to content at this point than it was to, to product. It was like, okay, some of the things that I was doing were getting really, really popular And I was watching, you know, whatever, Twitter followers, like just, you know, going, going north, like really quickly and, you know, big, big hits that were coming out and people were really having fun following this story over months as I was working on this freehand thing. And I think the, uh, I didn't have a plan beyond like, yes, I wanted to use this thing and yes, this tech was like, uh, an interesting thing to work on also, but I was, I was really starting to, to get into this idea of like, Okay, what does good developer content look like and why do people like this so much?

What do you mean by developer content? Sorry to interrupt.

Yeah, yeah, yeah. Like, uh, like social media content essentially, uh, which was being consumed at, at like, which was just bangers, like hit after hit for, uh, an audience primarily of software developers and engineers. And, uh, just, just basically the journey, like watching you go on this journey. Yeah. And, and again, I had tried to do this. At Framer and it like could not crack this nut like at all. Like, you know, a couple of little successes, but like it was incredibly frustrating. And so I'd always, you know, wondered, I'm like, what is, what is the actual, like, what do people actually want? Like that, what is that? What does that relationship, uh, like what, what would have worked and what, what would work in the future?

Try to crack this, meaning again, like almost building for an audience, like building for a social media audience. Yeah. Like what? Like you're almost like performing in a way. Yeah. Yeah.

It's like, and I think that the thing that I ended up with, the working theory that I had there was that like the majority of software developers that I know and that I talk to are working, not necessarily boring jobs, but, but jobs in which there are clear goals. There's a clear thing that you're building. And it is also because it is on Rails in terms of like being product managed or project managed, that there are always these really interesting unsolved problems that any developer, probably anyone in any industry, kind of like runs into along the way to say like, ooh, yeah, that's that. You know, how do you do that? How do you, you know, like And then you're like, well, that's not what I'm here for today. So I just have to like ignore that and, you know, like maybe file that away as something to do, explore later. Uh, and I think that the kind of the fantasy, the indulgence of something like Perfect Freehand, that what I was doing in my nights and weekends, you know, while working at Play, um, was that I had run into one of those problems. It was like, yeah, how do you do that? Like variable width line. In vector space or something. And I'm like, you know what, 6 months. I'm just going to keep working on this obsessively until, uh, until I figure it out and make it a tiny bit better, a tiny bit better. There was never any project, you know, even now as a, as an employer, like I wouldn't give 6 months to anyone to work on that problem. Uh, it was so small. It was just really hard. Um, but I was doing it. And I think that, that that idea of like chasing down a highly technically challenging but also highly like opinionated little, little problem and like seeing it all the way through. Like that's, that's a kind of like maker fantasy. That's the kind of like if I'm a mechanic, I don't want to watch YouTube videos of people who are doing normal mechanic stuff. I want to see YouTube videos of people like restoring a 1957 Chevy or something like that. Right. If I, if I do yoga, I don't want to see people who look like me doing yoga in the places where I do yoga. I want to see like someone in Bali I'm on a mountaintop, you know, doing yoga. There's a sort of like fantasy element to that. And I think that that's what I was at the time, like providing to, uh, I think that's why that content was hitting. Um, it's fascinating.

I feel like that's very different than most founder journeys, right? It feels like often people are building products that they want to see exist in the world, or they have an idea for a business. You're almost like you said, you're, you're solving these really hard problems for a very specific audience. Almost to, if I'm, if I'm hearing you correctly, to, to like kind of see if it can be done, uh, almost as an art. And which I guess makes sense given your, your background. Um, well, it's really fascinating and really cool.

Yeah, thank you. I mean, it is, uh, it was incredibly fun to see, you know, because these things were open source, anyone could take them and use them, you know, in whatever direction you wanted. Uh, you know, people would sponsor me and give me money, like, and these days I I also, when I see a new company using, uh, Perfect Freehand, you know, I go on Twitter and I say, hey, this is great. Here's my sponsor link, you know, uh, you know, sponsor me. And I think the last time that happened was, uh, was Canva. When, when Canva started using us, we, you know, I did that. I went on Twitter and I'm like, hey, this is, uh, you know, you should, you should sponsor me if you're using this. And, you know, Canva, whatever, $49 billion something revenue, uh, They could do it. And, uh, it ended up getting a really, becoming a really popular tweet, this sort of like, you know, hey, why aren't you giving Steve any money for this thing? Uh, and then they did, they gave me like, whatever, $5,000, but it ended up being like, uh, yeah, yeah. Perfect freehand. Um, but it's, uh, it was nice. And the fact that I could spend time on these things and then they would actually become useful and people, you know, part of products and, you know, make a minor improvement on someone's, uh, interaction with a certain feature or drawing feature. That was, yeah, like I love seeing people use, use the, the open source work that I've done. It's like incredibly satisfying.

Talk about this decision to keep Perfect Freehand, uh, open source. Like you, you talk about this, this call to action for Canva to sponsor you. Mm-hmm. Why, why keep it open source? I guess talk about that philosophy a little bit.

I think at the time, so I think my feeling on open source has moved a little bit now that I really see how valuable these types of things can be and to what kind of companies they can be valuable to. And also just how relatively easy it is to say, you know, yes, this is open source, but so you can learn from it, you can make improvements to it, you can, you know, and a lot of something like Perfect Freehand was possible because it was public, because other people could contribute to it, or read my code and tell me what's wrong and submit bugs and all that. So the, the community side of open source is great, and that's really why, why I love doing it. I think if I did something like Perfect Freehand again, like the default would be like, hey, if you're using this in a non-commercial usage, like, go for it. Uh, if you're using this inside of a commercial usage, you know, here, I'll sell you a little license to do so. Uh, turns out that's, that's pretty easy. Um, and that's kind of what I would recommend any ambitious open source project to do. Like, open source in quotes or whatever. Like, I think having, um, it's entirely acceptable and reasonable to say, um, there's a limitation on commercial use of my little code project that I made. 'Cause you never know. I mean, sometimes a $50 billion company will use your stuff in a way that matters. And yeah, a little part of me is like, ah, kind of wish that I had been able to capture that value more.

But that's a new perspective for you, or a newer perspective, it sounds like. You didn't always have that point of view.

It's not easy to do. It's not like there are obvious ways of doing semi-commercial open source for people. I think a lot of folks, myself included, just, if they don't think that a thing is terribly important or if it's not their job and it's not their company and you don't wanna worry about taxes or you don't wanna worry about selling or anything like that, it's just like, yeah, it's open source. But it is possible that things get really, really popular and you kind of wanna protect that eventuality should it exist. So how does Perfect Freehand turn into TLDraw? Right, so having built Perfect Freehand also involved building this experimental, uh, design tool called Globs. It's at Globs.design. Uh, I had this idea, it still might be possible, but it was like, all right, rather than, um, rather than describing a shape as like a polygon, I could describe it as this like glob shape with these two circles that are connected by two arcs. Um, it's hard to describe. I couldn't make it work, but along the way I built essentially a Figma-like design tool, right? Had a lot of the same interactions that you would have in any design tool. The only thing is, the only thing you had to work with was globs. That was the only shape. There was no rectangles or squares or circles or text or anything like that. It was just nothing but globs, which was fun. And I learned a ton from that. And Of course, coming off of that, I was like, and again, I had shared that in public and people got really into it. Uh, after that I was like, all right, well, that got pretty popular also. Uh, let me make, but like I wanted to add things that weren't globs to it. And so I was like, all right, well, let's make kind of like a bare bones, uh, canvas, something that, that looks and feels and has all the tools and interactions that you'd expect from something like a Figma or a Sketch or a Framer or a whatever, you know, um, but which can, can have like custom shapes added to it that like doesn't really care about the primitives. It just has everything else. And so for me, I expected to continue building these types of projects that use this type of visual canvas. I'm like, I want an environment in which to build those things and to experiment with them without having to start from scratch every time. Uh, and so I started out building that and got really into all of the, um, this is where it got started getting really popular because I was doing these, these threads and deep dives into these features that like everyone was using in Figma or everyone was using in Sketch or something, you know, that like the, the amount of, uh, complexity and feature set that we take for granted. In something like Figma is massive. Like every infinite canvas has like hundreds and hundreds of these like micro features that are each individually like really crazy complex. And they just keep on going and going and going. And every single one of those just, I was able to turn into like a really good Twitter thread and like in GIFs and, you know, show how it works of like, I don't know, how do you, what happens when you're, you're rotating like a group of, of shapes and like, You know, why is, why are they wobbling around and all this and get into the math. And I was again doing this very publicly and, uh, this attracted the attention of Figma, of Adobe, of Miro, of pretty much everyone in this space. Uh, well, this was that, that weird thing that came after Globs that I hadn't given a name to yet. Um, and I eventually, uh, did give it a name. I used that old TealDraw domain. That I had back from Telestrator. I was like, all right, I'll just call it TealDraw, easy 6-letter domain. Wonderful. I didn't even make that whole TL;DR connection until months later. But, you know, that's what I went with. And I got to a point where like, you know, I was working on this, I was really enjoying it. I wanted to finish it. I was either gonna take a job working on the Canvas team at Figma, or I was gonna take a job on the internal prototyping team on the user experience department of Adobe. Both amazing jobs, dream jobs. We were just negotiating like, you know, between the two. And so I was like, you know what, I'm going to, I'm going to quit Play, going to take a couple of months off. I have some contracting money saved up. These guys, I'll let the negotiations go and I'm just going to work on this, this, this TealDraw thing for a couple of months. And kind of get it out of my system before starting another job, because it was pretty intense at this point and taking up a lot of my mental time. I didn't want to start a new job while still working on this. Ended up going with Adobe and signing a contract with them in the summer of 2021. To work there? Yeah, to go work for Adobe. But I said, hey, I don't want to start until January You know, beginning of January, I wanted like a couple months off to, at this time it would be like 6 months off to finish this Teal draw thing. And they said, okay. And so I went full-time on this. And I think this was like the end of July, immediately picked up a corporate sponsor who paid me like $75,000 to have early access to the source code, which was interesting. I was like, huh, why would this very large extremely large company with a bunch of developers, like, not want to build this from scratch and want to build on top of my thing instead. They were integrating a whiteboard into a video chat application. And, you know, talking to them, it turned out that, like, well, this type of application is really hard, as I was learning. It's really complex. But it's also, like, incredibly subjective. And kind of like, just like our arrows or the perfect freehand thing, like there's no right way to build it. And there's no, uh, there's no obvious way to like solve some of these problems. They were just like, well, what feels good? What feels right? And what I didn't realize, but while I was implementing all this stuff, I was constantly making those types of subjective calls. Um, but with a background in design, background in interaction design, uh, you know, the big fuzzy background in visual art and sort of. Somehow everything added up to a bunch of, I'm not sure you could say like right decisions, but the type of decisions that some technical teams would have agonized over. And I was just banging them out left and right, you know, all day. And so it was not only like the, like, yes, it did the thing that it needed to do. And yes, the code worked and everything, but like, it was solving problems that not everyone would have been able to see as a problem. And doing so in a way that would have been, for most people, choosing between two identical, you know, tasting wines or something like that, you know, and, you know, you need someone with taste. And I had the taste. You had the taste. Yeah. So that was good. I went full time on it, started working on it. During that time, I got a lot of interest from other teams for the same reason, wanting to build on top of this and wanting to have early access to it. Ended up securing like another major corporate sponsor. This time it was— I was like, yeah, how about $120,000 for a sponsorship? Like, you get nothing, I get money, I'll talk to you like once every 2 weeks. And they said, sure. So cool. Wow. Basically, like a whole bunch of people were throwing money at me essentially to like stay unemployed and stay working on this thing because they wanted to build on top of it. They wanted to incorporate it. They wanted to evaluate it. They wanted to just compare against it because there really wasn't any good open source tech even to like learn from in this area. Like whiteboards are hard, but they're also like carefully guarded, proprietary, competitive advantage like tech. And I was just building it, you know, I'm like, yeah, I'm going to open source this thing. Like anyone can take this and run with it. And so that's eventually what I did. And end of November, I launched I open sourced all the code. I launched tldraw.com. It was free. You could go there, you could collaborate. And it got super popular. I had a ton of inbound from, I mean, I had a ton of users using it. It was top of Hacker News. It was top of GitHub trending. There was a ton of activity around the repository. People were really into this. But then, yeah, I had a ton of, inbound from other companies being like, this is great, we're going to start building with this. Or can you build these 3 features so that we can, you know, do this? Or like, hey, we've already started building with it. Or, hey, you know, 2 months, 2 weeks later, like, hey, we've just shipped this to production. And it was just wild, like so much attention, so much inbound. And I didn't know what to do with it necessarily. I— it was much more successful than I thought it would be. I had a lot of offers to sort of acquihire me in order to basically bring that into a company and for me to lead a team kind of building their whiteboard or something. And I also started getting interest from investors as well to say like, this doesn't really happen as often as you think it does. There could be actually This is more pull than most startups get, you know, like, have you considered? And so after talking with a lot of different people and kind of going fairly deep with a couple of different avenues, I ended up deciding, okay, you know what, I am going to raise capital. I am going to do a startup around this. Number one thing was that like what people were asking for was way more than I could deliver alone. And the types of companies that were asking for it made me think that actually it was, this could be something that got big. And my other big, you know, thought and one of the things that I had like kind of been talking about, not 100% seriously, I hadn't really taken it as seriously as I may have, maybe could have. But essentially that like this type of application like the whiteboard felt like it was going to be much more common in the future than it was at the time. That's not to say that like whiteboarding apps were going to get more popular, like things like Miro or Framer or whatever. It was more like, like a collaborative whiteboard is going to be a feature of many, many, many apps that don't have it currently. Right. It's not going to be a product. It's going to be a feature. Exactly. It's going to get— like there was a time with like Trello where like a Kanban board was a Trello board. And like if someone, you know, if you were dragging cards between columns like that was on Trello. Now every app has that interaction. It's just called a Kanban board. It's not called a Trello board. And that it just became like a commodity. It just became like a common feature, a type of a thing. And I had a feeling that like one collaborative whiteboard probably should already be that. And I didn't really understand why they weren't until I started working on TealDraw and I was like, oh, that's why. 'Cause it's fucking impossible to build these things. Like it's incredibly hard to build these. And that if I made it easy for people to do that, then they would.

So when you go to raise the money then, and do this full time, is the goal, like you said, there were a lot of things that were being asked of you. So you wanted to hire and you wanted to be able to deliver on them. But was it also maybe to go back to this point around, you know, this realization you previously had that, you know, maybe open source should be commercialized in some way. Was it also to go and kind of like search for a business model?

Yeah, I mean, it was, I think by the end of the race, I had a decently compelling picture of like, here's what I would do. Here's what I want to build. But I think the pitch that I probably could have made was like, look, I don't know. Like, I think that there is more pull for this idea that like, people want to build these types of products. but it's not worth it to build it from scratch because it's very difficult and it's expensive and it's whatever. But if we can get them almost like all that undifferentiated stuff solved and sell them that, then, then this thing would, would become much, much more popular, uh, that we could create this sort of category of, uh, uh, of applications which are not solely dedicated to being collaborative whiteboards, but which include collaborative whiteboards or whiteboarding-like things. Inside of their products. And that could be open source, it could be not open source, it could be a service, it could be a hosted service. I had no idea how to like, you know, what that right way of positioning it or selling it or anything like that. And I think where the future is gonna be is collaboration is much more popular, is that the whiteboard is not seen as a, or the canvas is not seen purely as a graphical thing, but as a productivity space. That people want websites that just sort of feel like Photoshop or feel like Migma or Figma. And also that like Figma's popularity was getting so big at the time that like every, like every product owner is gonna grow up on this software. People are just gonna want things that feel this way. And then around that, I forget exactly when the sale was announced. I think it was after we started the company. But that, that only accelerated the thing is just like, wow, this is, this is going to be like people, people realize just how much users love working with this type of interface. Yeah, we just want to be there in the future ready to sell this thing.

But I imagine one, you probably never could have predicted what GPT-4 Vision could do for you. No. Or maybe you could. I don't know. Tell us a little bit about what happened there. I mean, that's, that must be a fascinating new direction for the company. For sure.

So I'll compress the year and a half that happened between then. Basically, we just started building it. I put together a wonderful team here in London. We kind of started from scratch again, solved, you know, basically building. And I had to call Adobe and say, hey, I'm not going to start on Monday. That also happened. How'd they take that? Well, you know, kind of the same way that a player they had taken it, which is like, you know, well, I'm happier that you're not going to go work for a competitor, right? Like, instead you're going to go work for yourself. And, uh, yeah, it was very supportive. Um, the— that's good to hear. Yeah. Uh, we had, by, by the middle end of this year, I mean, we were, uh, we had a, a new whiteboard that was incredibly good, that was getting popular, uh, that people had already hacked and built not hacked, but just sort of built with and built upon to build a bunch of production apps, even though it was in like alpha releases, including some larger companies and a more successful startup. So that was the path that we were kind of going down, getting ready to do our kind of go-to-market and all that. And then suddenly GPT-4 with vision came out. And previously, the only thing that you could do with GPT was essentially use TealDraw as the output. You could say, hey, ChatGPT, and we had played around with this. We never really shared it because it was not super impressive, but you could say, hey, this is my API of my canvas application. You know, draw a Christmas tree. You know, and it would go like— you could do this before GPT-4 Vision? Yeah, yeah. Is that like—

how are you doing that?

Well, you know, it, it's like it has a kind of a concept of like what a, what a, uh, triangle looks like, you know? Okay. If you say like, here's a, here's a function called create triangle, um, and here's my coordinate system or whatever, it can make 3 triangles that are on top of each other that are or slightly bigger, like, and in that case, what was happening was the input was text, you know, into the AI model, and the output was text back that we were using to draw inside of TealDraw. But essentially TealDraw, the canvas was the output place. It wasn't the input place. The input place was still text. What ChatGPT-4 with vision did was it let us use the canvas as the input rather than, what is this Christmas, draw me a Christmas tree or something. You'd say like, hey, what is this thing that I just drew? Describe this picture to me. And it could. And then you could say, okay, well, how many ornaments are on this Christmas tree? And it could count them all. In the case of MakeReal, it was like, here's a a bad drawing, a wireframe of a website. Hey, GPT, you are a web developer. You've just gotten this drawing from your designer. It's your job to make an operating, like a working prototype. Send me back the HTML that I would need in order for it to make like a single page working prototype. And it worked, which is amazing. Wow. So the first version of this was from a guy named Sawyer Hood, who was actually a designer at, or is a designer at Figma, and literally like hours after GPT-4 with vision was announced, they produced like a prototype using TealDry that did exactly this. You drew your thing, it sent a screenshot of that to ChatGPT-4 with a prompt that said, whatever, you're a developer and blah, blah, blah, give me back the HTML. And then it put that HTML that came back into like an iframe. Iframe is like a website inside of a website. Yep. And then you could click on things and you could, you know, if you did your drawing right and said, uh, you know, um, here's a number, make this button, you know, make the number go up or something, then it could. Uh, the fun part was that you could also like, yeah, you could draw the thing you wanted to see, but you could also just like annotate it. You could do arrows and say like, in the same way that you would mark up a drawing that you were going to send to a real developer, should you be limited to sending only paper instructions. Yeah. You know, you'd be like, you know, this is what I want to see. I want to like this and then draw an arrow and like do a little bit of notes on the side and have like, you know, multiple figures that describe an interaction and all that. You could send that and it would come back and it would work. So we saw this and we were like, oh my God, you could make this like so much better. And so we did. We forked it and we started adapting the work that Sawyer had done. Into, into something else. Uh, the first thing we did is because, well, because our canvas in TealDraw is a normal website, like it's not a— most whiteboard canvases are like essentially images that are being repainted like every frame, like a video game or something, right? It's like a custom rendered environment using WebGL or WebGPU or, or any of these other technologies that, um, allow for like high performance rendering in the browser. That's what Figma does, et cetera. So in Figma you can do a lot, but they had to build it all. Like it doesn't, you can't use CSS inside of Figma's canvas because it's not a website. It's not HTML. The browser isn't rendering it. Figma's rendering it. But because with TealDraw, our canvas was a normal website, it still was in HTML land. We could put that iframe right back on the canvas. So what would happen is like you draw your little app and you say, hey, make this thing real. And it sends it off to ChatGPT-4 with vision. Like, here's a picture, here's this prompt about you being a developer, give me back the HTML. We get back the HTML and now what we do is we create a shape on the canvas which has that iframe, which has that website like inside of it, and we put the HTML there. And now, I mean, you can still rotate it and resize it and move it around, whatever. But it's like back on the canvas. You've never left the canvas. Hmm. And the real breakthrough for us was that now that it was back on the canvas, you could also draw on top of that, uh, and say, you know, hey, this button that is green, like make this red or, you know, cross out one of the elements or something like that. And then you would select the, the, the website along with all that annotation and use that as a new input. So you'd click that Make Real button again. Ah. And now we would send it a picture of the website along with its previous HTML, its previous output, and the screenshot that we've taken of like you marking that thing up and use that as a new prompt to say like, hey, it's like, it's fascinating. It's like your designer had notes. Yeah. Yeah. Yeah. It would, it's like, uh, the, the prompts this time would be like, Hey, your designers have just come back to you with some notes on your previous output. You know, please incorporate these annotations and stuff into a new output, you know, a new HTML file. And then we would get that back. And what this allowed was— and you could just repeat this, you could just keep going, you could iterate and iterate and iterate. And it was mind-blowing because this was like a workflow that I had never experienced. No one had ever experienced in anything. This ability to iteratively add complexity to and shape the, like, deep, deeply shape the behaviors of a little tiny prototype using nothing but, like, arrows and lines and text. But you could also use, like, screenshots from other applications and say, make this look like this. Or, like, you know, like the notation, like crossing things out and other like, typesetter type of markup. Um, and it worked and people were sharing this stuff like crazy. Of course, like, this was going mad on Twitter. Like, yeah, no, I saw we were doing like Kanye West numbers in terms of views and likes. It was, it was just going everywhere. Uh, and, and like, yeah, I'm sorry to anyone if I just took over your timeline for, uh, for a couple weeks in November. Um, but it was like legitimately something new. And the things that people were able to make, you know, with their kids or, you know, with their classrooms or just themselves, like adding, you know, they, what was fun is that they weren't just sharing the output, although you could, you know, you could, it gives you a link, you could open it in Replit or open it in CodeSandbox or just share that link and actually use it. It's like permanently there. Um, but what people were sharing, what weren't like the final result, they were sharing like the journey. They were sharing all the steps, the iterations, the markup that they had written, the, this, you know, like that became the story of like, here's what I did, but also here's how I did it, which is something that you just don't get. You don't get that with code because code doesn't work that way. Uh, you don't see all those previous iterations of your code at the same time. You don't get that with design. Sometimes you kind of get that with design, but like, then you don't really have a working final result. And it was just, it was incredibly magical. People are still using it a ton today. And what's, what's even better is that because for us, this was like, this was like a demo, this was an experiment, we just tried to stay on top of the popularity as much as we could. But, you know, we never charged for this. In fact, we couldn't even have charged at the time. The GPT-4 Vision APIs were so new that you, you essentially had to— you still do— you have to enter your own developer API key in order to use it. So you're using your like, which is bad for a website to ask you to put in an API key. Don't, don't ever do this, but you can do this with the Tildraw with MakeReal. Yeah, you were just being charged directly from OpenAI for using your own API key. When I eventually did talk to OpenAI, they were like, yeah, how is this so popular? Like, because the rate limits are really tight. Like, you can only make like 100 of these requests like per day per API key. And they said like, are you doing account sharding? Which is where a developer would open up a whole bunch of different accounts and then just sort of like route a request randomly to one of the 5. So that now you have 5 times as many open. And I'm like, well, no, we're not doing that, but we're sort of doing that in that we're asking for a number of users. Yeah, we have a number of accounts that, that each user has created. Uh, we had to do a ton of QA, or like questions and answers, uh, community work around how do you create API keys and all that stuff. And yeah, that's probably—

that's not an easy thing to educate on for people that aren't developers.

No, it was not a it was not an end-user product. But what's cool is that people are now making actual production end-user products around that same pattern using TLDRAW as like the engine, the input device for working with AIs and generative AIs. And that's really like, that's the place where I think that, I think we would much rather be is essentially the like providing that visual surface for working with these types of models. Um, cause you, yeah, you know, it's not text, like text is, it's easy to write a text box. It's really, really hard to write a, uh, a canvas. And, um, it just so happens that we, you know, we had a canvas that was really, really, really ready to go right at the right time. Like I've said a couple of times, what we ended up building with TealDraw, like I hadn't already built it, I would start building it because it was, it so clearly fits this world in which, you know, you have, you have to be able to, to not only like compose images, but like iterate and work with agents like in a longer term way on a canvas. And, uh, yeah, incredibly fortunate as a startup founder for, uh, history to like make a hard turn into your trajectory. Uh, got lucky there.

What's so fascinating to me about this is in many ways it's like come full circle back to the early days when you were learning Framer. And getting deep into prototyping, you know, TL;DRAW, like you said, through this hard turn in history, in a sense, has turned into a prototyping tool. But actually, it sounds like you have a vision that it could be much more than that. This could be a platform whereby people actually are developing real applications and deploying them beyond just prototypes. Is that the aspiration now? That like that is what Make Real is going to be? So I don't know.

It's a good question. I think with Tildraw, the— there's always been a tension between are we a company providing like a developer tool? Are we Mapbox, you know, or are we Google Maps? This is the open source question from earlier.

Yeah.

Are we, are we providing a tool to other businesses to build upon? Mapbox I use as an example all the time. If you have like a food delivery app and they have a map in it, like it's Mapbox, it's Mapbox. It's like no one, no one builds maps. Maps are insanely hard to build. I wouldn't build maps ever. Um, but people like it, uh, and people do it and they, they, uh, they sell it and they license it essentially to other companies to, to use inside of their products. Um, and that has been the direction that I, that TealDraw has always been down. Um, but along the way there's, there's been a couple of times where we're like, you know, this is kind of getting pretty popular as like an end user app. Maybe we should just do that. And so we've had to re-decide multiple times that, like, you know, what kind of company are we? Are we— what kind of product are we building? And it always just seems like the, the more interesting place to be is on the sort of Mapbox side of things. Is that like We could create a good prototyping tool that uses a canvas in order to be the input device and be the iteration method for, for building websites or products. But, you know, like, you could also use that same canvas to build, to do backend stuff, you know, to edit and create databases. You could also do it to, um, you know, take screenshots of applications and, and mobile applications and say like, hey, actually move this button over here, you know. And essentially like the, the pattern that we've found with Make.Real, I think, is a, is an incredibly generic pattern that could be applied to 1,000, 10,000 different use cases. Um, like it could be one good product Uh, but I think if we did go that direction, someone else would just make that generic thing because it's just totally impractical that all 10,000 of those, um, those companies, before they get to make the amazing AI, you know, controlled HVAC, you know, draw your HVAC heating and ventilation system and we'll, we'll convert it into the actual, um, uh, schema or whatever, uh, before you get to do that, you have to figure out how to resize a rectangle when it's rotated, you know? Right. It's like no one wants to do that. They just want to start at the point where all that stuff, undifferentiated stuff is done and just go from there. And yeah, I think the— we'll probably continue to experiment with Make.Real. We're certainly still continuing to experiment with ChatGPT in a couple of different ways. But my guess is that the function of Make.Real will still be to inspire other developers to, to build products, you know, and buy TealDraw in order to, to do it. Uh, but to that, it's, uh, that it's more of like a juicing the imagination, uh, of other developers and, and proving that like, wow, this really connects with people. Like, imagine if we use this inside of our educational app. Imagine if we use this inside of our, you know, org chart thing. If you could just like cross out the guy that you just fired and have that. You know what I mean? And it's true.

It rewrites the whole org chart. Yeah, yeah, exactly.

And if you can draw it and communicate it in a way that another person would understand what you want to get out of it, either GPT will already understand what you want to get out of it and create the same result, or it's like 6 months. You know, and then it will. So I think just having like a really, really, really good, flexible, um, extendable web-based canvas is like an incredibly well-timed product. Um, if I hadn't made it, I would be super envious of whoever had made it. Um, and yeah, so we are, uh, we are in the business of demos and we're in the business of a, of a kind of an infrastructure product, essentially a UI infrastructure product. Uh, and people are already building really great stuff on it that use the same Make Real pattern. Um, so it's exciting to see.

You, you couldn't have predicted this, it sounds like. Um, but maybe if you were to look ahead, what other things, uh, might be coming down the pike in terms of AI that could impact TL;DRAW and your path moving forward?

I think the biggest thing will be, uh, kind of AI collaboration. Okay. Essentially the same experience that you have collaborating on a whiteboard with someone else, another human. If you can imagine that the other cursor on the screen was being driven by a model instead. And that in the same way that you can have like a kind of a, basically the ChatGPT moment for Canvas is coming. I know Steve Ruiz has experimented. A real-time thought partner. Yeah, yeah. That you're not only like, not only doing things like a kind of an AI-based autocomplete, you know, like I start drawing a graph and they're like, ah, I think I know what the rest of this looks like. Let me suggest it for you. But to have a sort of a visual conversation about like, okay, here I'm planning a trip, you know, here I'm, Here are a couple maps and here's where I'm gonna stay and here's where, you know, where else I wanna stay. Here's my road trip route. You know, pull in some suggestions for places we wanna stay along the way. You know, how would you change this route? Or like, here, I'm gonna start drawing a layout for my apartment, you know, and I want you as the cursor to sort of be, to be making suggestions, to be modifying things, to, Essentially have that conversation with me. It is, I think the canvas as the place where multimodal prompting and interaction and iteration happens seems pretty natural to me. The real question is, can we get the speed of that communication and the multimodal aspects of it like fast enough and correct enough and teach it to use the product in a capable enough way that it really does feel like you're, you're collaborating with someone else, even if that someone else happens to also know the location of every Burger King between New York and Chicago.

Like, you could totally see that happening. I think it's a great prediction because, I mean, today if you think about ChatGPT, it's, it's it's more or less what you're saying, but an asynchronous experience. Um, but if the thing could move fast enough that it could be synchronous, the type of workflow that you're talking about seem— would seem totally possible. And that would be— that would be awesome. Yeah.

And if we jumped on a Miro board, or a— or a TealDraw, uh, for that matter, um, you know, right now what we would be doing is we'd be using texts, we would be using visuals and screenshots and drawing things. We'd also be using our voices to communicate. It's like everything at once. And I think that's why people like collaborative whiteboard so much is that it's fast and it's just like stimulus dump of information that you have like multiple channels going at the same time. And so you're able to communicate really quickly. You even have this notion of presence and closeness and proximity and all that stuff. And yeah, much, much more complicated than just a chat back and forth. But I think that's going to be a direction that people want to want to go. If you really want to maximize the type of information that you're able to exchange with a model or with an agent. Yeah. Like it's going to be on the canvas. And my hope is that we continue to kind of stay on top of things so that we are the place where that experimentation happens and where those products happen and that we're able to get all those people started without having them learning about, you know, how to intersect an ellipse with a second ellipse. When those rotation— that's terrible. Uh, all this stuff that is undifferentiated will be taken care of. You just start at the, uh, start at the end and go from there.

Well, I, I, for one, I'm very excited for this prediction to come true. Um, and, uh, yeah, we, we've covered so much, uh, and it's been, it's been fascinating to hear this story. Is there anything you feel like we haven't touched on that we should? I don't know.

I mean, there's the, uh, the normal startup stuff. Uh, we're based in London. We're, uh, we just raised, I mean, we had our seed round, uh, which was led by Lux and Amplify and a whole bunch of really great angels. Um, and we just did, right before we got super popular with the Make.Reel thing, we had just closed an extension, which was another $2 million. Yeah. Thank you. And, uh, So that was led almost entirely by Amplify. So great group of partners there. And we're based in London. We are in unusually, like we have a big office, or at least an office that we all use. 7 out of the 8 are here in London. So we're building that type of company. And yeah, if you are, if you're in London, if you're into this type of thing, Definitely give me a shout. You know, we have more desks than we need. You can work here for the day.

Where can people do that? How, how can they get in touch with you?

The best way to get in touch with us is either on Twitter, we're very active on Twitter, uh, or x.com/tealdraw. Um, we also have a big Discord where you could join if you have questions about the products or if you want to use it. Um, again, we license it kind of in the way that I described before. If, if it's, uh, Non-commercial use. If you're experimenting, it's free, go nuts. If you are using it in a commercial product, then, you know, I'll set you up with a license and it's pretty good terms and all that. But the— yeah, that's the direction that we're going with it. And we're small, like we're 8 people. And yeah, it's a— it's an interesting time.

Interesting company. Fascinating. Steve, thank you so much for your time. It's been so, so incredible to hear the story and we really appreciate all the time. For the insights you've given to us. Thanks so much. Awesome. Thank you. Thank you so much for listening to Generative Now. If you liked what you heard, please do us a favor and rate and review the episode on places like Spotify and Apple Podcast. It really does help. And if you wanna learn more, you can follow Lightspeed at Lightspeed VP on YouTube, Twitter, or X, LinkedIn, or anywhere else on social media. Generative Now is produced by Lightspeed in partnership with Pod People. I am Michael Magnano, and we will be back next week with another awesome conversation. Thanks so much.