Building Scalable Web Automation Infrastructure | Josh Coyne & Paul Klein IV

Foreign.

I'm Paul, founder of Browser Base, and I started my career as an intern at Twilio, which was kind of like my KP fellows experience. So, like, all of you are at, like, this very formative part of your career where you're probably like, what do I do next? Should I go join Clulee or should I go, you know, start a company or do I work somewhere? I mean, these are big decisions that you're making, and it's so commonplace to think about. Like, the transitional time in your early 20s is like, so big. So I've been there before. You might have all the answers yet.

That's okay.

But the people you meet now are the most important things. So this room is great. It's full of awesome people. I know two great people here from my company. Talk to those guys. They're cracked. But Browser Base is really kind of like the extension of my career arc. Where I started as an intern at Twilio during their ipo, which was actually my first open bar, and also my first ipo where I had no equity at all as an intern. A crazy place to start. But through that journey of working at Twilio, which is an API for sms, and then starting my first company as cto, selling that to Mux, which is a video API company, I really found my, my, my niche in developer tools and infrastructure. I am a developer myself. I love building software. And for me, building this company, Browse, which is also an infrastructure company, it's. It's really fun because I get to build stuff that I would like to buy. And I think that kind of mission of like building things that you would use, it really does resonate. It's kind of cliche, but it's true. At browserbase, we're building tools for AI agents and people to automate the web. We're building a web browser for AI, which is just a programmable web browser that you can control with code. And LLMs are fantastic generating code. So with browser base, AI agents can now go to websites, extract data, click buttons, fill in forms, download files. All this work that we do online and the future of software is software doing work on our behalf. A lot of the work we do in a browser browsebase helps power that future of software. So you look at it, it may look like, oh, I started this company because it's an obvious idea, but actually it's an extension of my time working in infrastructure companies. My passion for headless browsers or browsers running on a cloud that I learned at my last company and when you put the story together of how I got to where I am now on paper, it makes a lot of sense actually. And this probably is the only company I could have started. Like there's nothing else. And it's worked very well, probably because of that. And I think through this conversation and through talking all of you later, as you craft your own personal narratives of your growth in your career, thinking through what's the next logical move that makes most sense, that tends to be the one that's going to have the best results because your experience does shape the work that you're doing in the future.

Awesome. Super cool. So a lot of these guys, I imagine either now or later will want to think about starting their companies. So let's take you back to the beginning of browser race. You say, I have this aha moment. I incorporate, maybe I build a little bit of an mvp. What do you do next? Like, how do you get your fierce users? How do you learn from them? How do you do that kind of iterative cycle initially to make sure that there's a there there?

Well, I was actually, before I even started the company, I was really trying to convince myself not to start a company. I spent. You've read the original memo I wrote. I wrote like a 3,000 word essay about this space. And I wrote it because I had this idea of like, it seems, seems to me that like the browser and AI being able to control a browser will be a really important primitive for AI agents. There's this great Andre Karpathy blog post about the llmos and the browser is that bottom square there? And I knew a lot about headless browsers from my time at Stream Club being cto. And I really wanted to make sure that this is the right thing. Because when you start a company, and especially if you raise venture capital, it becomes a multi year commitment, right? You're going to be doing this thing for a long time and you don't want to give up. That's the worst thing to do. So I really want to be sure. So I did like 50 interviews and I just started like searching on Twitter, searching on Reddit, talking to friends. If you live in San Francisco, you meet a lot of people in startups, you can just ask them like, are you guys working on this problem or have you heard of this problem before? And after, like the 50th customer interview and like the fifth time a customer said, if you build this, let me angel invest, it became pretty clear to me that there was something here building better infrastructure for headless Browsers running in the cloud. And from that, I got to go back to the people who I interviewed and say, hey, like, I actually ended up building this. Do you want to come, you know, try it out and be a design partner and early customer? And I kind of just kept doing that. I kept meeting people in startups. I'd go to, like, every meetup ever for the first, like, three months of browser based. Like, I was at so many AI meetups, which there's a lot in San Francisco. There was maybe a little bit more in early 2024 and just talking to everyone who would hear me out. And a lot of people wanted to try this thing. And at one point we put a video out on Twitter that got maybe 100k views. But it really started the flywheel of people starting to hear about browser base and coming in via word of mouth. And word of mouth has been our strongest channel since its inception, because if you can get people to remember your product and it comes up again when they hear their friend talking about it, they're going to come give it a try. So as much as you can building that initial awareness of the thing you're building, and it might not directly lead to a customer in that first conversation, but they may be talking with their friend and their friend might be complaining, like, oh, my God, I'm dealing with headless browsers and puppeteer and Selenium, and it sucks. And if you did your job right and you're lucky, that original person you talk to might say, well, I actually met this guy from browser base. Have you heard of it? And that's the flywheel of the word of mouth that helps grow these businesses, especially in the earliest days.

Super cool. And if you were to talk to yourself a year and a half ago, as we think about, maybe I'm gonna call them painful decisions, but maybe things you do differently. Like, would you have changed anything of how you started browser based and how you built for the first three, six months?

No, absolutely not.

Wow. Okay.

I think I nailed it. Like, the way I nailed it was I had so much, like, prior work before starting it that, like, the path for the first six months was actually, like, abundantly clear. And I think that's the threshold. Like, if you look at, like, great companies like Stripe, like, they kind of knew from the beginning what they were going to go build. Supabase, like, same thing. Like, especially in my category, avoiding that early pivot is actually critical to success. The companies that don't have an early pivot, I mean, there are pivot Success stories and they're always incredible. And those people have insane perseverance. But the people who don't pivot, especially if they want to win a category as competitive, if you can kind of get the first like six months right, you're gonna have a lot more likelihood to succeed and you can build your momentum much easier and you have a lot more willpower too. So I think a lot of the prior research, you know, I mean, of course there's stuff I would have done better and stuff I could have done better, but I actually think it was all really meticulously planned out up including which investor I wanted at each stage of the company up until the Series A was pretty much picked the day I started it.

Well, and to Paul's point too, if you think about like Stripe's payment infrastructure, Supabase database or browser bases, headless infrastructure like this is going to go on. These businesses alone will compound for hundreds of millions of dollars in revenue. So it is a really nice foundation to build not just like a platform but like a really large company. And if your initial product is not going to do that, you know, pretty quickly usually, and hopefully that's something to find out earlier than later. Can I ask maybe personally, professionally, like what has surprised you most about being a founder?

Yeah. And maybe on that prior question, one second, like, sure. You have to like pretty much if you're going to build a venture backed company, decide that from the beginning and make sure the idea is going to be venture backable. Right. So that was part of it. Like I knew that this thing, if it was, if the early customers were right and it was all going to work, then I think we could go pretty far with this.

Right.

But what was personally surprising for me, and I think people don't take this as serious as they should, is it is taxing. And people always say that it's like, well, Paul, whatever. I think I gained like 30 pounds the first year browser base. Like I've never seen a Series A founder with a six pack ever. Like, I just at one point I like, I've run three marathons, I've done a full Ironman, I've climbed like V7 at the climbing gym. I gave all of that up for this and my 2024 goal was obsession. Like I was going to be obsessed with this thing. And you know, guess what, you'll survive if you don't go to the gym for a couple months. And yeah, maybe Brian Johnson's like your lower I, you'll be okay like a few percentage points Off IQ is fine. Like you, if you're going to win and you're going to be a founder, that you have to go all in on this thing. And that going all in is very hard. You know, I didn't see some of my friends for a long time and I, I basically hung out with my wife and I worked on browser base and got in a bike ride every once in a while. And that, that's like the degree of obsession, I think. I really love the story about Bill Gates where, I think I've told you this, where, like, there's a story where he took the radio out of his car and then an investor was driving with him and they're like, why'd you take a radio out? He's like, if I had the radio, I wouldn't be thinking about Microsoft while I was driving. I'd be listening to the radio. Like, that's psycho. And I'm not doing that. Like, I still have my AirPods. But I think if you look at really, really successful founders, the level of obsession is something to be admired and you don't have to go too crazy. You know, you have to make it through the journey of the company. So don't, don't kill yourself. But there's probably like a degree of obsession that's a little further than you think is obsessed with that you should try and strive for, to really go the distance. And that's. It's a big challenge for sure.

And maybe for some of the fellows who are thinking about an idea, is there a right time to go and kind of jump off and build that startup? Or is it like, what's, what's your reaction to that?

Yeah, I mean, I think I have a hot take here. I don't think you should rush to start a company. I meet a lot of young founders and they always feel like, well, I got to do it now, otherwise I never will. And I think that can't be further from the truth. Like, there isn't a timeline. I mean, unless you have something that has to be built right now, there will be many great startup ideas in your time, and if you start one now, you're actually probably missing out on the opportunity cost of being at a great company that's working and seeing what product market fit is like up close, seeing what a company is like when it grows or gets its first customers, and rushing into it actually could result in you burning out from ever being a founder. And what I talk about at Browserbase is like, my goal is that after working at our company hopefully your future startup, it's gonna be a few percentage points more likely to succeed because you saw what it was like when somebody had product market fit and how you go from 1 to 10 million in revenue or 100 to 1000 customers and working backwards from that. That is the invaluable lesson, I think, because if you haven't experienced PMF before at a company, maybe you worked at a big tech company that's too far away. You have to be pretty close to it to see it. You won't know if you have it or not, and you'll spend a lot of time iterating on ideas that might not ever be viable in the beginning. So your pattern matching is actually worse if you rush into it as opposed to building up a little more experience and starting the company.

Awesome. So one of my favorite Twitter presences, besides maybe the CLUly guys, is wpaul@browserbase both because there's both insightful tweets, but also really fire tweets that make me laugh. But one of them is actually a talk he gave that was awesome called the web browser is all you need. So Paul, what's the gist of that and why do you feel like MCP and A2A and all these other aging frameworks are insufficient?

Yeah, I'm not anti mcp. Well, I think I've come off that way a little bit because I was just a little critical of the framework. But I think that when you're building an AI agent, your AI agent needs tools, right? We build the browser tool for AI agents and there are going to be amazing MCP servers you add to your AI agent. But you probably can only add like a finite number of tools at one point. Like dynamic tool discovery is still early on and fitting that all into the context and having select the tool accurately is challenging. You need a few like primitive tools. So you probably like executing code. You probably need like file management. And I think you need a browser because if your user is asking your agent to do something and there's not a tool you already have built in, the browser is kind of like the default tool that we would all use. You know, it's like if you, if you're building a travel agent, the travel agent agent, and you ask it to hey, can you book me a ferry from Santorini to Athens? And you don't have a ferry, you know, tool, what's it going to use? It can use the browser. So I kind of feel like it's like this essential tool in the MCP ecosystem, but I think there's a lot of stuff to be figured out with mvp. But regardless, whether you're using our framework stagehand to control a browser or you're using MCP1 of MC servers and having a browser in your agent is something that you probably want to have as a catch all tool for building agents that are working in production.

Yeah, my reductionist take on this one is it was like when the APIs came out and everyone's like, okay, no GUI that will never exist again, we're going to do all API. And it's like, whoa. Of course that's not the case, especially for agents, as we're saying. And maybe to that point this probably gets to director a little bit, but I think we're starting to get closer to human level performance on online web tasks. What do you see today as the biggest limitation of that? Or do we feel like you've kind of already hit sufficient parity to do most tests?

Yeah, that's not even my craziest take on Twitter. We can do that one. Next, I Are we close to human level performance? No, not at all. I mean if you look at like the best the state of the art evals. Well, first of all, a lot of the benchmarks and evals for web agents or AI agents that control websites, they're not sufficient, they're. They're quite bad. Like web. The web Voyager eval set is if you look at the data, it's like really weird and not very well representative of what people do. So I think we're lacking great eval to compare what is even performance. But I mean, how do you say they're good enough? Are they as fast as people? No, certainly not. Inference is still slow for reasoning models, are they as reliable? Yeah, in some scenarios, yes. It definitely depends on use cases and for what we tell our customers is that they say, well, are AI agents browsing the web good? I say it depends. If you're just using Craigslist, they're fantastic.

Right.

If you're trying to have it play Doom on a browser, like probably not as much so. Right. So I think there's still a lot more that's going to get better with web agents. But for, for certain tasks they do quite well and especially they do well if you can define your steps quite clearly. The biggest limitation for agents is actually prompting. Right now I think models are quite good if you have a very detailed prompt, but getting users to prompt your web agent the way you want it to is quite hard. We've spent a lot of time Trying to build tools to make it easier for people to prompt web agents and get reliable results. An example of this is like you might ask it, hey, go book me a flight. And you're not saying, is it business class or first class or economy? Is it a one way? How many people are on it? Do you have TSA pre check? What's your number? Like, these are all things you don't put into a prompt. And your agent might fail to do the job 100% right. Because we don't want to tell it all this context that we have, but if you do, it does a pretty good job.

Cool. And for those who don't know, browserbase has this amazing framework called stagecam that helps solve a lot of this stuff. And then, please, we should talk about director at some point. But maybe last question for me, then I can turn it over to the audience. Q and A. We're moving really fast in AI. It almost feels like it's hard to. Every week something changes and you have to kind of go on Twitter and spend a couple hours to figure out what's the latest. But in your eyes, what's the most promising area of research today, especially as it relates to human level performance in the browser? And do you feel like it's kind of just a matter of time or if we need new architectures or different compute models? I don't know. I'm just kind of throwing stuff out there. What's your reaction to that?

I mean, I'm probably the most like unscientific AI CEO. I'm an infra guy, you know, I really focus on infrastructure and dev tools. But I had a really insightful conversation with someone today which was like one predominant theory of like making better models that can browse the web is you like spin up a bunch of fake websites and you do reinforcement learning on those fake websites and the question we had is like, does that generalize well? Like does the reinforcement learning or the post training on the data that we have generalized well to the whole Internet? How do you teach something to use the whole Internet without seeing the whole Internet? Right. And I think generalization and web agents is still like not yet fully understood and determined. So I'm excited for that. Like, I feel like we are still super early on the model side in the way that we try and approach that at browser base is like, well, with the models we got, how can we use engineering to make them work really well? Okay, well, we can basically say you have to define the steps that's part of our framework or hey, we're going to use multimodal screenshots plus DOM because that gets better results. Like, I think you can apply engineering to where the models are now and still build amazing products. And then what happens? It's kind of nice. The models get better and your product gets even better too. You can remove some of the engineering. You make it more simpler over time. So even like the prompt engineering hacks of 2024 aren't as necessary context have gotten longer. Like I don't think I haven't used it in a vector database in forever. I don't know if you guys have live coded something recently. So it seems like as things continue to get better, there's less engineering required. But at least for now, you can, you can engineer your way into something working really well with AI on the web.

Awesome. And before I turn it over to folks, actually, can you talk about Director AI?

Yeah. Okay, so my ask, thank you all for coming to my talk. I hope you get some value. But I have one ask. We just launched a new thing. It's called Director AI. It's basically like a version of OpenAI's operator, but after it does the task, it gives you the code that you could then import into your IDE and run that task again infinitely. So if you would all give it a try and just send me some feedback Paul at Browserbase, or you can click the little feedback button, that would make my day. And I think there's like what, 30 people here. I'm going to look for 30 emails later and I'm going to check the roster and we'll have a fun happy hour for that.