Faster vector search with Turbopuffer CEO, Simon Eskildsen

Hey y'.

All, welcome back to the show. This is Database School. I'm your host Aaron Francis and today I have Simon Eskildson on. He is the co founder and CEO of a very cool company called TurboPuffer. And TurboPuffer is a serverless vector search, full text search, interestingly built on top of object storage. And so it's built on top of Amazon S3 and as we get into in the interview, also on Google Cloud and Azure Blob storage. The interesting thing about this is it makes it obviously extremely scalable but also way, way, way cheaper. And so he talks about some of the numbers in this episode of the number of vectors they're doing it is insane. And they're weirdly. Their first customer was a little company you might have heard of called Cursor. I don't know how that's their first customer, but this is a great show. He's very, very technical. I think you're very much going to ENJ. Please subscribe to the YouTube channel or visit the website@databaseschool.com and let me know what you think. Enough from me. Let's get to the show with Simon. I am here today with the co founder and CEO of TurboPuffer. His name is Simon Eskildson. We're gonna talk, we're gonna talk about a lot of stuff. We're gonna talk about scaling, we're talking about founding companies. But Simon, do you want to introduce yourself a little bit further?

For sure, yeah. I mean, you did a good job today. I spend most of my time working on a search engine called TurboPuffer and building the company around it. I came of age spending almost just shy of a decade working on infrastructure at Shopify. Pretty much anything to make sure that the site stayed up and stayed scaling for all of the big customers and the tremendous growth of that company through the 2010s. And then after that, I helped some friends with infrastructure challenges at their companies kind of in little three month stints. I called it angel engineering. I took equity for improving postgres query plans and optimizing Autovacuum for a few years. And that's where I discovered the problem that eventually led to turbopuffer. That the incumbent search engines were not a delight, which was exactly my experience at Shopify. Now I've run TurboPuffer for about two and a half years alongside a fantastic team here. And we're very proud to power many of the companies at the frontier of AI.

Wow, that was tight. That was really well done. Okay, so I want to talk about. I feel like there was this glory days for Rails engineers and it was that time you're talking about like the early tens and it was like Heroku, GitHub, Shopify and everybody was on Rails and it was the. Y' all were the cool kids and I was so jealous. So I want to talk about.

What were you doing?

Were you like lost in the PHP found? First of all, I was found in php. I was not lost. I love php, but I was not the cool kids, Unfortunately. Actually in 2010 and 2011, I, I just graduated college in 2011 with my master's in accounting. And then I spent one year at Ernst and Young being a tax accountant, so. So in that sense I was super lost. That part sucked. But yeah, I missed that.

But PHP found you at Ernst.

PHP found me. And that I do love. But I missed out on the cool San Francisco Rails companies which are now, you know, all billion dollar plus companies and we don't have many of those in the PHP land. So that's what I want to talk about. You were there at Shopify for I think eight years, scaling from, you know, thousand requests a second to, we'll say a million. You know, that's at least orders of magnitude correct. So was your experience like there and like what did you come in as? Just like a Rails developer. And then what did you end up as? And what was kind of that journey through that hyper growth phase?

That's exactly right. I started, I mean I, I started, I started in PHP too. Is. It's so good. The feedback loop is so good. I still miss it, you know, creating websites for the World of Warcraft guild. And then at some point you're, you're just, you're writing the same thing a million times. You build your own framework. That's how we, that's how it all ends. You probably did that too. So how to build your own PHP framework from scratch. And then this, this dude in my World of Warcraft guild was like, have you looked at Ruby on Rails? And I'm like, you go to the website back then? And I was like, is this like, is this a toy or is like what is this? And was very intrigued and then got into Rails. And I spent a lot of time with Rails in, in high school, worked for a company in high school, a startup in high school doing Rails. And then that's Shopify found me because I wrote this article about dropping my iPhone and it just died. You know, the, like the 2010, 2009 iPhone, you dropped it once actually. It's funny, I think about it, I dropped my iPhone this morning and it didn't get destroyed. But back in 2009, if you dropped an iPhone once, it was dead, right. Screen smashed, it was game over. So you guarded that thing like an absolute. Like it. You know, Otter boxes were a thing.

Otterbox cases. I remember that.

Yeah. And I didn't have one of those and I just dropped it and it got destroyed. And then I wrote an article about how I suddenly had my sense of directions back because I wasn't looking at a map all the time. And it was like I had an attention span and it just like sat and looked into the ether and it was actually pleasant. And this was before we had realized all these pernicious effects of smartphones. It was just not a thing. In 2010, and this article blew up and. Or, sorry, 20, 2013 and this article blew up and Shopify reached out and I don't think they realized that I was in high school until I. Until I told them that I. That I. The start date has to be after the. After the semester. I was like, aren't you? Anyway, so I started there in 2013 as. Yeah, a Rails engineer. And I sat down to lunch one day and I was so intrigued by what the infrastructure people were talking about. It was this, like, load balancers and, you know, squid here and varnish there and like nginx and, you know, I don't know, they probably use them all together. Even though that was crazy and I didn't understand, like, what's a proxy? What's a reverse proxy? I don't think I still know what a reverse proxy is. I don't know if anyone knows why it's called that, but this was very intriguing to me and I just started picking up stuff that was in their issues and just started doing stuff with that team and then somehow like found myself there working on Docker and, you know, 2013 was a crazy time to get into Docker and got into that, got Shopify running in containers and that was sort of my foray into infrastructure from. From Rails.

Yeah. It's funny that you talk about, like, in 2013, you dropped your phone and the world opened up before you. It's like, how quaint. Like, dude, can you imagine, like, in 2013 looking forward to 2025 and just like realizing how much we're all going to be on our phones? And you had that realization a decade ago, like, man. And another thing I want to highlight is something I talk about quite often is, like, you really one not you specifically. One really needs to be writing and putting work out there. And even if that's, you know, tweeting or making videos or posting on LinkedIn, God forbid. And like, your story is very similar to how I got my first PHP job, which was, I was doing something and I wrote about it on my blog and some company reached out and was like, hey, we're doing that too. Can you come work for us and we'll pay you. What to me was like a huge amount of money. And I was like, the Internet is amazing. And so it's nice to hear. It's nice to hear stories where you put something out there and good things come your direction. That's really, that's really encouraging for me.

I feel like another way to describe my teenage years was as a series of blog posts that made it to Hacker News, even if it was for a brief period of moment of time. That's how I got my first job. It was how I went to Shopify. It was how I got my second job. It was. Well, actually the job I had in high school was because allegedly one of the co founders was looking. I was, I grew up in a small town in Denmark and they were looking for. I should do this. I need to take a note of that. They were looking for people who are committing on GitHub. You know, again, just, just committing to GitHub in 2010. Super strong signal. And committing to GitHub in 2010, 2009, between the hours of 2:00am and 4:00am in this small town. And they somehow found me through that. That was their talent sourcing process. I need to do that. You go, you go to, you find to like, you go to like some tiny town in Nebraska and you just zoom in and you just geo filter into that and you just find Alex. And Alex is just up and it's 3am and she's just slinging C code because she still hasn't heard of Rust because no one in the town has

even heard of C. That is a great story. Yeah, that I feel like, I feel like back then GitHub, you actually like followed people and it was a social network and then there were forums where people would actually like hang out and talk and you could really like find some undiscovered talent in those places. And I don't, I don't know if that's still the case. But hey, it's worth, it's worth looking into. So when you were, you started as a Rails engineer, talk to me about some of the scale that you Saw maybe in the second half of your career as you transitioned over to the infrastructure side or started getting really good at postgres and like what kind of scale growth did you see at Shopify while you were there and what were some of those challenges?

Yeah, when I started working on infrastructure at Shopify it was a couple hundred requests per second and the last flask sale before I left was, was around a million requests per second. And on a, on a Ruby on Rails cluster, this is quite sizable scale. And so I worked, when I worked on doing that, my time there was just making sure that everything in the core app and all the core database stuff just, just worked. And we were a team of comrades on the last resort pager there, there was about six to eight of us and we were just dealing with whatever came up. And some of us, I was often working on the more longer term database projects. Make sure we could run in multiple regions, make sure we could fail over a region with a chatbot and do that with very little notice. All these kinds of things of just sharding everything as hard as we could on the Shop ID key. That's what I worked on for a very long time. Everything, caching, everything database. Because that's the hardest part to scale when you're scaling something as quickly. As we scaled Shopify, it was a journey of a lifetime. I also worked there, I worked on search, I worked on me and Justine, my co founder. We rewrote the entire Shopify storefront and then build a wonderful team to do that for performance and all the things that we'd learned about about running Shopify. We got a rewrite to serve almost all traffic in 18 months for all of Shopify, even though the app is almost 20 years old. Just so many fantastic things we built nginx and nginx Lua were just the secret weapons to run all of Shopify, to shed traffic. So that when Kylie Jenner was selling lipsticks that, you know, Aaron, I don't know if you have a Shopify store, but if you have one, Kylie should not be able to take it down. Right. And prioritizing traffic, multi tenancy engineering. I would never have guessed how much this would translate into starting a database company, which I always felt incredibly insecure about because I haven't written a database. I've used databases for a very, very long time, but most database founders have built databases their whole career, so. But I've mostly just used them and bought them and looked at a lot of websites for them. I think our website is sort of the antithesis. Because a lot of the times when I go, when I went to a database website when I was at Shopify, I felt like half the time I couldn't figure out if they were selling some new cool bespoke fashion product or if they're selling a database. And I just really, I just wanted to know, like, what are the trade offs you've made? What are the guarantees that you provide? What does it cost? How is this to run? What's the architecture like slotted into my mental model? And that's. Those are the questions that I hope that when people go to our website they just feel that are immediately answered.

Yeah, yeah, I want to get to, I want to get to Turbopuff or what it is. But just right before we do that, how fondly do you look back on your time at Shopify? Like when you look back on that, what are the feelings that you have?

I think the feelings are there's a video of me that my dad has from when I was a kid where I just got off a roller coaster at Disneyland Paris and say. I say something along the lines of, that was really fun. I never want to do that again. I think that's, I think that's the feeling that you walk out of a like hyper growth generational company run if you have really just given it everything you had. That's how I felt. And when, when I stepped away from Shopify, I, you know, we were, it was, it was, it was. There was some recovery to be done. And I don't think that I ever thought that I would step back into the ring and the fire again. But once you've experienced it, once you just, you end up chasing it for the rest of your life. Even if it's with some periods and different seasons of life in between.

Yeah, no, I totally resonate with that. I mean, that's how I felt after our first set of twins was born. And then we had a second set. So I get it, I get it.

That was two sets of twins.

Two sets of twins, two and a half.

You have four children and you at some point had two children at two and a half and two newborns.

Two two and a half year olds and two newborns.

I don't know how you were here today.

Never wanna do it again.

Yeah, I don't know how you're here today. That is more remarkable than what any of your guests have ever told you on this. That is epic.

Yeah. Needless to say, I resonate with that feeling deep in your soul of man. That was awesome. Boy, am I exhausted. So I get that. So you come out of sh and you spend this, these three months at a time, you know, going around and helping friends, you know, taking, dusting off some of your war stories and taking them to other startups. And then how long did that period last where you're doing this consulting? And then we'll get to Turbo Puffer.

Yeah, I think first, first I was just like this is the summer where we just get the deadlift up as much as possible. And it was just like the summer of Simon. And I loved it. I was just writing napkin math posts and paddleboarding like a maniac and working out five times a week and it was glorious. And then by the fall I started getting bored with that and that's when I started just sort of in three month increments joining my friends companies and vesting a bit of equity in return and helping them with infrastructure stuff. And the common problems around then is basically just at Shopify we use MySQL so if you're using MySQL in the 2000s, you feel incredibly gaslit by the orange site that you made the wrong choice in 2005. Doesn't matter how good your business is, it could be better if you were using Postgres. And so I was very excited to finally get my hands on this thing. And it turns out that it's just a completely different set of problems. And one of the trade offs is tuning the auto vacuum. That's what I spent a lot of 2022 doing, was tuning Autovacuum. And one of the companies I was tuning Autovacuum for was this company called Readwise. And Readwise, what it is, it's that it's a way to read articles and books and whatever later, highlight it, retain it, search in it. And they asked me if I could build a little recommendation system. And I'd heard of these things. I mean, I don't, you know, I'm like a database nerd. I don't really know anything about recommendation systems. So I just started researching and I found these things called vector embeddings, right? Where basically you, you feed an LLM a bunch of content, you chop the head off and then out comes this like coordinate in a coordinate system and magically like coordinates that are similar adjacent in the coordinate system are also of content that is similar. And that was really cool to me. And so I read a bunch about that. How do you build these models? And, and so what occurred to me is like, well, these models are Actually, these embedding models are trained on the articles because they're trained on the public web. So it must be really good. And so I built a very small recommendation engine over the course of a week that would sort of like take articles that you've read and then, you know, with vector embeddings, find other articles that were similar. And it kind of worked like it was kind of, it was kind of interesting. And it was at least good enough that I got recommendations from one of the team team members about articles that were around like pregnancy and so on. And I didn't know that, I didn't, I didn't know at the time that his wife was pregnant. But it sort of leaked, it leaked through the article recommendations because, you know, so it was at least that good. And. But the problem was that, you know, sort of I got something fine, wasn't excellent, but it was promising enough to go in this direction. And I ran the back of the envelope math on bringing this to production on one of the vector databases at the time and it would have cost 30 to 40 grand a month. And this was a company that was paying three to four grand a month for their postgres and maybe a couple of other thousand per month for all their other infrastructure. So you're almost looking for them to spend an order of magnitude more to power the infrastructure for one feature. So it sort of entered this bucket of we'll do this later. When did costs come down? And that problem haunted me because I looked at the pricing. Well, clearly they're doing everything in DRAM and they're replicating it. It's like there has to be a better way. And so I maintained this, this repository. Have you, have you seen The Napkin Math GitHub repository? Have you ever encountered this? It's essentially just a really shitty rust script that I wrote that figures out memory bandwidth, disk bandwidth, like all these different sort of base numbers. And it's very useful because you can do things like, oh, this algorithm is taking four seconds, but actually with the, you know, it's traversing 20 gigabytes and you know, you can do that in a second on a modern machine. So what's this 4x gap? And you figure it out. This is, by the way, this is way better method than profiling. Profiling just sort of like is for iterative improvement. Napkin Math is very good for finding order of magnitude improvements. And I just started doing the napkin math on building a search engine where you put everything on S3, everything like no, nothing else, no raft or paxes or consensus bullshit. Just everything on object storage. And it seemed like you could actually do it. Like, the fundamental trade off was that the right latency would be higher. And then once in a while, when you queried something before you loaded it into cache, it would be a little bit slow, but you might be able to get it off. And so I just. I fully, like, locked up in, you know, at my cabin in rural Quebec for the summer of 2023 and just forced through, trying to find a way to make this architecture work. Because it felt very clear to me at the time that if I was not going to do it, then someone else was going to do it. And there was just these, like, S3 had only gotten the right primitives to be able to do this very recently. NVME SSDs were recent. Ish. Right. And it felt like the perfect moment and the perfect workload to do this for. I couldn't articulate it very well at the time. It was like, there's something here. And I was just like, trying to explain to my wife and she's like,

I love you, Simon. I have no idea what you're talking about. Yeah.

And I. I released it in. In October and. And it was like. It was like the fourth rewrite and I was going manic and there were so many things it didn't do, but it just needed it out there. Like, at that point, it was just like it. Here it is. Can you swear? I feel like I've sworn a lot. Is that cool? Yeah.

Everyone's an adult here. I think we only have 30. 30 and older probably listen to this show. So it's fine.

It's dropped it on X and it got. It got some good traction and. And people were encouraging. So that's sort of. Sorry, no. Was not open source. I didn't have time for open source.

It.

Yeah, it was a product. It was a SaaS product. Like, give me your vectors, I'll give you the results. It was. It was. It was just that. And it got really good traction that encouraged me to keep going. And then a small company in San Francisco reached out and they were having trouble with. We're paying way too much per user to store all these vectors of code bases. And I had never heard of this company before, but for whatever reason, something compelled me to go visit them. Probably it was because they were so busy that they kept not showing up to our scheduled calls. And I get it, like. And then. So I went to San Francisco and had a great chat with them and they decided to adopt this little database called TurboPuffer, this little innocent thing. And of course that company was cursor and yeah, just very special relationship. And anytime I got the opportunity to, I would pass on as much knowledge as I had about tuning the postgres auto vacuum or anything that I'd learned from Shopify to them. They don't call me anymore because they're probably better at it than I ever was now. But it was a really fun relationship between two very early companies. There's a lot less early than I was at the time. But they really took a chance on TurboPuffer and they really took a chance on me and Justine, my co founder. And as all of this was transpiring, I was just thinking, who would be the best person that I know that I could do this with. And Justine just immediately came to mind. She was the one that I'd worked on so many incredible things on at Shopify. Everything I'm proud of there, and even projects that neither of us are proud of, we've worked on together. And it was just. She was just the first that came to mind. And I've never seen her do the things that she's now able to do. It's incredible.

And so this is an amazing story. And so you all. So you and Justine, is it the two of you, y' all are the co founders.

That's it, yeah.

Okay. So you and Justine founded this thing, launched it a couple years ago and somehow either the first or the first big or one of the first customers was Cursor, which is insane, by the way, just insane that that was one of the first big ones. So backing up just a little bit, you're seeing, you've seen in your experience, people struggling with search, recommendation, relevancy, that sort of stuff. And you're seeing these primitives come online with S3 and thinking these two things, these two things can meet. And it seems like there's something that should be done here. And you go off into the cabin and write it up and it works. And so I want you, I want you to give me the elevator pitch for TurboPuffer. And then I want to talk about what you discovered about S3 and kind of how you architected this whole thing such that you can make this work. So what is turbopuffer?

Turbopuffer is an opex storage. I mean, depends a bit on who we're talking to in the elevator here. Right, so we'll hit it from a few angles here. Turbopuffer is an object storage first database. It is a search engine that allows you to connect enormous amounts of data to AI. Examples are Cursor, right? And Notion Linear and many, many others. And the reason they choose TurboPuffer is because we can index and connect more data to AI than they were able to do before. So Notion used to have a per user AI cost, and when they moved to TurboPuffer, they could take that away and they could index more data than they had before. Cursor used to index into thousands of files before TurboPuffer, and now they index as many as they can find and the list goes on. And we help our customers realize the most ambitious versions of their products.

Okay, I love it. So a couple of things stood out to me in that succinct pitch. One is Object Storage, obviously that is, that is unique. And the other is you said both database and search engine, which don't always go together. Right? And so talk to me about, like, fundamentally when you were starting this, what was the thing that allowed Object Storage to work here, where maybe three, four, five years before Object Storage was like, no, that's never going to work. What was it that you saw about Object Storage that changed your mind?

I'll explain this in a way that I like to think about it now in that if you want to create a big database company, you need two things. The first thing that you need is that you need a new workload. If you want to create a really big database business, you need more or less every single company in the world to have a use case for your new database, or at least consume one or more products that have a strong use case for your database. That's ingredient number one. If you want to build a big database company. The second thing you need to build a big database company is that you need a fundamentally new storage architecture. Because I can promise you that if you have a new workload, all the other database companies are going to look at that new workload and say, I want that. And there's no good reason why they shouldn't get it and fragment the market. Unless you have a fundamentally new way to store the data that they can't rewrite everything to do because it would, it would screw up their guarantees and trade offs. The new storage architecture is to store everything on Object storage and then puff it up into NVMe SSDs and DRAM as you query the data, almost like a JIT compiler. That's the new storage architecture. It is only possible now because of three things that have changed. The first thing that's changed is that NVME SSD'S are available in the clouds as of around 2017 is when they launched in AWS clouds. And even then there are very few of them and very few SKUs. NVME SSDs are about 100 times cheaper than DRAM, but only around 5 times slower if you use them correctly. Using them correctly means that you have to have a lot of outstanding requests for every round trip, right? So you can't just use them like you use drammers. Like I want this and I want this and what this in random order. You have to say I want these one terabyte and then I want this other terabyte and you have to do all of that in parallel requests. And the storage engines had not been built for that. The second thing that changed with storage architecture is that S3 only became consistent in 2020 at re invent. So that would have been December. This is completely overlooked because everyone just assumes this. But that is not that long ago, especially not in database land where it takes a long, long time to mature. The third thing that happened that's very important is that S3 only became only had Compare and Swap available as of December of last year, like remarkably recently. And what Compare and Swap allows you to do is to not have a consensus layer laterally to your database to decide what keys hold what information, all the metadata, but to do that directly on S3 so that your metadata can live in a file called, you know, metadata JSON. For all we know, you download the file, you do the changes, and then you upload it and you know that it has not been changed in the interim. That means you don't need another metadata layer. That means that you can build a database where you just run everything on stateless nodes and the only state is in object storage, which means it is more cloud native than any other database architecture that we can think of. And it's also fundamentally the cheapest way to run a database in the cloud. Those are the three things that changed that allowed a new storage architecture. That's what means that the economics and the underlying architecture is fundamentally different than MySQL, than snowflakes, than databricks, than all, than Mongo, than all the existing databases. Those are the two things you need.

That is very helpful. Describe to me this, you said you puff it up. Describe to me that part there. So where is it long term? And then what is this process by which maybe you can do request lifecycle, you can do search query, whatever example you want to use, but describe this process that the data goes through to fulfill the user's needs.

Yeah. So what happens when you do a read? You will basically go to a load balancer and the load balancer will say, well, what table is this for? We call this a namespace. The namespace is logically just a directory on S3, right? So one could be called Aaron DB and one could hold SimonDB. It could also be, you know, notion, customer N, or it could be cursor code base Y. Right. These are the prefixes. We call these namespaces. It's a new database construct. So if I am querying Aaron db, then I am going to the load balancer. The load balancer will/Aaron DB and then send me to node 2 out of 128 or whatever. It will look at the DRAM cache and see if it's there, and if not, it will go to the NVME SSD cache. If it's not there, then it will go to Opyx Storage and get the blocks directly. TurboPuffer's database engine is completely optimized for OPX storage, to the point where when we get data from the database, we do a range request directly into the S3 file of exactly the data that we need, because we know where it is in the file. A lot of databases that have tiering will basically download an Entire file from S3 hydrated into cache and so on. But we can operate directly against S3. When you have a cold query that misses DRAM and misses DSSD and goes directly to object storage, the latency is around a second, 500 milliseconds to a second, depending on how hot the S3 Shard is. If it's in disk, it's often less than 100 milliseconds. And if it's in memory, it could be around 10 milliseconds. So it's sort of like these are the order of magnitudes of where the caches are, right? 10 milliseconds, 100 milliseconds in about a second. In practice, it can often be faster, but that's a useful way to think about it.

How do you know when you're going to. Let's say you fall all the way through and you got to go to S3. How do you know which range of that file to get? Like, who told you that? And second question, what is in those files? What. Like, what is the data that you're storing in those files and how are you storing it? Like, are you breaking it up? Is it, like, compressed? Like, what does that actually look like in there?

It might be Useful first to talk about turbopuffer v1, like that thing that I hacked up at the. At the cabbage in. At the. Not cabbage, at the cabin in Quebec. And then, and then, you know, brought, bought, brought poor Justine in to help optimize it. Before we could get to V2, V1 was you take all the vectors and you cluster them. So should we do a little detour here into what a vector index is? Is that what this looks?

Nothing more. Yes, please.

Perfect. The simplest way to do vector search is that you. You have a. Well, first, maybe I'll explain even what a vector is. So we got a little bit into this. But you know, I was feeling, I was feeling funny before in the way that I explained it. So we'll do it properly now the way that I usually explain it. And you should ask follow ups here because we're backing into a long winded answer to your original question. A vector is a point in a coordinate system that represents a piece of data and the point is adjacent to other things that are similar. Right. I'm standing in front of a table. So the table would be the coordinate system here and right next to table would probably be chair, but closer to table might be dining table. Right. So you imagine that you train a model that is very good at taking content that is similar and putting it in. In the coordinate system, if it's a really smart one.

Clarification. So I'm in a. I'm imagining an XY coordinate system because I am a human being and I can understand three dimensions.

You don't have a space frame.

Two dimensions pretty well for sure. Two dimensions. And so I'm imagining, you know, table is at 1:1 and dining tables at 1.1. 1.1. Right. So that of what I'm imagining in reality, as I understand it, you can choose your dimensions and you could be not a two dimensional, not a three dimensional, but a seven, 60, a 10. It can be huge numbers of dimensions. Is that. Is that accurate?

That's correct. So the. I just explained it in two dimensions because I don't have a big brain enough to imagine a 768 dimensional.

No.

I would think maybe if the two sets of change that twins that you

did, but all brain power's gone.

You might be. Do you want me to explain this in one dimension?

Yeah, that would be awesome. Okay, so when we're saying dimension systems, we'll talk about 2D or 3D. Understand, dear listener, that it's many, many, many more.

Exactly. Let's just like just, just to really Hammer it home. Imagine Spotify. They have all of their songs in a big coordinate system. And you can imagine a rock cluster in a pop cluster. You zoom in and you find little small clusters in there. That's how a simple, the simplest form of a vector index works. You basically cluster it. So you find patterns in the, in the coordinates. And then you say, when I'm searching for a song, I'm just looking. You look at the clusters that are similar. You basically take an average of everything that's inside of a cluster. So the average of all pop songs and the average of all rock songs. And you search with your query vector and say, well, it's closer to the rock, so I'm only going to look at the rock songs. And Suddenly you're searching 50% less data.

So your universe got a whole lot smaller.

Exactly. And that's how you build a. That's essentially how you build a vector index. There's a lot of challenges with that. We could get into that, but that's the simplest way. So what Turbopuffer V1 did was that it basically took all the data and then it built a bunch of clusters, rock, pop, hip hop, whatever. And then it created the file called centroids Bin and it put that on S3. So Aaron DB so who.

Where was that? Like, let's say that I am, I'm giving you all of my music. Am I locally creating vectors? And then I'm throwing vectors over the wall in v1, am I throwing them over the wall to you? And then you cluster them and centroid them or like who's doing what where?

That's right. You are just like, this is like stripe for vectors. You're just sending a vector and then you can search the vectors and that's really all you can do. There is nothing else really to do. So you send all the vectors and then turbopuffer takes all the vectors and it clusters them and then it creates. Then it takes the average of every cluster and uses that centroid and puts it in centroid JSON, we'll just call it that. It wasn't actually JSON but just like. So it's simple. So Centroid JSON is basically just an array of arrays, right? Where every array is sort of like rock and then acentroid, pop, centroid, whatever. And then we have another file that's called cluster 1, JSON, cluster 2, JSON, cluster 3, JSON and the centroids map back to that. Now the way that you serve the query is that you go to S3, you get Aaron DB slash, centroids, dot, JSON. You look for the closest centroids to your query vector, and then, you know, okay, well, you know, this is sort of in between, like, pop and country. Whatever. There's probably some artists there, I'm sure, plenty. And you look up that was cluster 79 and cluster 108, and you download those two files, you search through those, and then you return the results. Two round trips to S3 P90 to S3 is maybe around 200, 250 milliseconds. Corey is done in 500 milliseconds.

Okay, so I think I see where this is going, but I'm not gonna. I'm not gonna jump ahead, so let me say it back to you to make sure that I have it right. So I, as the user of TurboPuffer, I create vectors however I want, and then I send them over the wall to you and I say my namespace. Or maybe you define the namespace, doesn't matter. But you put them in a namespace of Aaron's database, and you create from all of my vectors, you kind of create like a. Like an overview, like a world map of the clusters, and maybe there's a hundred of them. And in centroids, JS or JSON, you have a listing of all hundred of those centroids. And then I send a query over that's like, hey, I want to find more artists like Taylor Swift. And you go to centroids with Taylor Swift and say, that's pretty much pop most of the time. And then you go grab the pop pop music JSON file, pull it back, and then search through the pop music to look for other artists that are like Taylor Swift, and then send that home to me.

That's right. That's right. Perfect. Yeah.

Okay.

Yeah. So you don't know about Centroids JSON. You don't know about cluster 123. Up to 128 JSON. That was more or less the one. Okay. More or less.

And it worked.

It worked. It powered cursor, it powered notion until the end of 2024.

There's a lesson in there. Okay? It worked.

It worked. And it was. It was. You know, Justine did all this crazy stuff because then, you know, I got, like, you know, I. I got ripped into B2B sales, you know, everyone's dream. And Justine was deep in the code mines, just, like, making all the shit code that I'd written zero copy and all this craziness to just make it perform. Then we hired. Then we hired some people who actually knew how to build Databases. And I mean Justine, just by her remarkable mind, knew how to build databases a lot faster than I did. But we started working on V2 in the spring of 2024, so about six months after launch. And this is where the sort of search engine versus database comes in. V1 is very much, it's very much a search engine because it can only search, it can't do anything else. V2 was a database that is excellent at being a search engine. A search engine to me is an attribute of a database, but a search engine is not necessarily a database. To me a database is something that in the limit could do any SQL query in a great way, right? Has a query plan for that. So V2 is like a proper LSM. It has a key space. We implement all of the vector indexing and this clustering and so on on top of the key space, but underneath Turbo buffer is just a KV store and then we build all this stuff on top. The LSM engine is optimized for Opix storage and all the trade offs of that which are different than doing it in traditional architecture. And that's how we, that's what we would call the database. Now of course you can't implement every SQL query in the span of six months on a new KV store. So it's very much focus of TurboPuffer to be excellent at search. But over time TurboPuffer is supporting more and more queries. Right? We have aggregations, we have group by all these different things that we will do over time. But our core remains to be an excellent search engine at indexing enormous amounts of unstructured data. So now we can do of course full text search and all these other things.

Can I ask why a full text search makes sense to me that's search realm. Can I ask why you did all of those other things? Because that sounds like a lot of work, first of all, and it sounds like search was the original thesis and very, very hard. So why add on this other thing that is also very hard? Who was asking?

So what you ask for when you have a search engine, basically when you have your data in a database, you start to want to just do anything with that data that's in the database. So I mean we get asked for all kinds of questions and very complicated. Probably someone's going to ask about CTEs and recursive CTEs at God forbid at some point, right? So people will ask all of those, but of course you have to roadmap it. And the reason that you want aggregations is because you want to do things like, well, I kind of need to know how many things are in the database. Right.

And so that's, that's, I buy that one. Yeah.

It's like, okay, that's useful. And then someone is, someone says, well, you know, it's really nice on an E commerce site site when you see, when you can toggle the sizes and you can show these, the facets in the sidebar. Well, that's a, that's a group by count. And then you know, you're like, well, I kind of want to know how many matches are in each document because I'm searching for these chunks into document or pages and I need to, these are aggregation. So it's not, it's not like people right now are saying like, hey, I want to do my, I want to do my revenue reporting on TurboPuffer. It's that you have this data in TurboPuffer and you want to do very reasonable things that are still in the area of search. And so those are the kinds of queries that we are prioritizing, but over time there's going to be lots of them.

So is it fair to say that you implemented a lot of this standard database stuff to solve the business search need, even though it is not the technical search need? Does that distinction make sense? Like as people are doing the business side of search, faceting is a great one. And faceting for the listeners, when you're like, I want red shoes and then like Nike drops down to 50 pairs and Adidas drops down to 25 and you're like red and size 12. And then it's like Nike has two pairs. And so you're kind of recalculating all this stuff on the fly. That feels like the business domain of search. But under the hood that's the technical domain of like, well, that's just kind of more traditional database. Is that fair to say, that distinction there?

I think so. I think it's just that people just need certain, you know, I think of all queries as SQL queries and there's just a bunch of SQL queries that you expect out of a search engine. It may not look like SQL because if I ship SQL to TurboPuffer right now, if you, Aaron, were using it, you would just get really annoyed at all the things that we don't support.

But all of CTEs and recursive CTEs.

Yeah, recursive. I mean you can do anything. Recursive CTEs, like these things are a godsend. I've implemented a spreadsheet, recursive CTEs DuckDB is very good at them. I think someone really nerded out over there. They're very good. I don't think MySQL even does it.

They do, they do in at least in eight. I don't think does them, but I'm

a five, seven guy, you know.

Yeah. Yeah.

So I think that there is just a lot of things that people expect from a search engine and some of those look like more traditional queries.

So that explains to me a little bit of the difference between V1 and V2 of scope and kind of like, I guess, thesis or mindset. But can we go back to the naive S3 implementation on the first one where you just had to do two round trips every time? And can we update that knowledge for V2 on how that looks and what V are we on? Are we on V2 right now or we're on.

We're kind of on V3 in some ways.

Okay, so you can jump to V2 or 3. Whichever.

Yeah. I mean, 3 is just 3. I, I would, I would just talk about V1 and 2, but we just started talking about V3 because V3, we had a customer needed to search 100 billion vectors, and that's puffing pretty hard. And so we need, so we need to make some adjustments to fit that scale. Suddenly I start to understand. Anyway, so V2 is an LSM. And so in an LSM, you're a little bit more civilized around these things, and it's not so tailored to one use case. And so in V2, what you do is that you, you have a, you have a key space, right? It's a, it's. And so you might have as key, you know, cluster. Instead of cluster 128 being a JSON file, cluster 128 is a key. And so you have to figure out where is that key, right? And so you download some metadata file, and that metadata file has an idea of what files contain what ranges. I'm really trying to simplify here. Right? But maybe you download LSM JSON. This is not how it actually works, but it's like pretty close LSM JSON. You get that. And then it says, well, in these files, these are the files. And maybe some metadata around what key ranges are in what files. And then you go to file, the file is just like UUID JSON, it's like some big file somewhere, but it's not a JSON file anymore. And you do a round trip to that file where you get the index block, and the index block is the last, say 32 kilobytes of that file file, you download that in another round trip. So you have two round trips now,

and that was a specific range request to just get the last part.

Okay, Just let, just get the last, like 32 or 64k. I can't remember what the block size is. It doesn't. If you're fetching 32 or 64k, there's barely a distance other than in the tail latency. You get that, and then in that you can see, okay, well, the keys are in these blocks. And then you do a third round trips. So V2 is actually doing an extra round trip because it has to get the metadata of which files are relevant. Then it gets the index blocks for those files, and then it gets the actual data blocks in the third round trip.

Why that decision? Are those files so big that it makes more sense to just get the index, calculate what other range you need to grab from that, and go back and grab that's right back.

Those files can be gigabytes large.

Gotcha. Okay, that makes sense. And then, so as you've described it, we're still just operating with, you know, turbopuffer is going straight to S3. And so then do we put those other stops in the middle and you just have intelligent caching along the way so that you're not grabbing whatever was formerly Centroids JS off of S3. You're not grabbing that off of S3 anymore. That one's probably hot all the time. The metadata is probably there all the time. And so you can just get that directly super fast. And then the caching layers come into play. Is that correct?

In the simplest form, I mean, the caching is ever evolving. And the smarter the caching we make it, the better, the better performance our customers get. Right? You can imagine that caching. That small LSM JSON is really useful to have in cache almost indefinitely and hydrate because you saved that first round trip for a very small file. So we could prioritize things like that in the cache. But in the simplest implementation of the cache, you can essentially just imagine that the NVME ssd, we create one file that's the size of the disk. You can imagine just as, as, as cache. That's just the name of this file. And you just, you just put the keys in and you. And you just load it in as a ring buffer. So it's like, you know, you get 1 gigabyte, you put that at the beginning of the file. Not a gigabyte, put it at the Beginning of the of, and you just do that and then at some point it wraps around, right? You just start overriding from the beginning. This is a very great way to use a modern disk because you're just writing as fast as possible. You can do lots of heuristics where like when you wrap around, right, and sort of the. It starts eating itself, you could start to say, well, actually this is axis a lot, so let's not override that. You could do all these kinds of things, right, and make it arbitrarily complicated. But TurboPuffer, like Turbo Buffer, just shipped in the beginning with just the ring buffer. It didn't do anything smart. And I think Justine had just had a 200 line implementation of this and it was rock solid.

That's awesome. That is so cool. I love a simple solution that just works all the time. So what are the. You talked early on about the turbopuffer website, or rather you talked about the fashion websites of databases in the tens, right? And how they just like, don't really tell you what's going on. Like, you just look sexy. So what are the trade offs of turbopuffer? What are the decisions y' all made that you want to like, illuminate? Because obviously it's very, it's a very different like mental paradigm than something like MySQL Postgres, SQLite, anything like that. So talk to me about those trade offs and what, what decisions you made to make this search case actually work.

The biggest thing is that turbopuffer has very high write latency. If you write to a MySQL, your write latency is essentially whatever an fsync is. On a modern flash drive, you could do an fsync in a couple hundred microseconds, maybe a millisecond. If it's a slow disk or network disk, that's pretty fast. And MySQL and Postgres and so on, you can even get it faster than that because of variety of things that they do. But generally it's sort of in the hundreds of microseconds ballpark. TurboPuffer is two orders of magnitude slower than that. Our writes take in the low hundreds of milliseconds because that's the latency you get to S3. If you were building the Shopify checkout with hundreds of milliseconds of latency on every write in the checkout path, it's just not going to work. Like you're just not going to buy anything. And that's the fundamentally biggest trade off that we make. The Other trade off, which is more of a trade off with the current implementation of TurboPuffer than a fundamental trade off, is that occasionally you will get a cold query. Occasionally, right. You will do a query and it will go directly to OPIC storage and then puff into cache. That doesn't have to happen. Right. We can guarantee that there's just always hot on at least one or two nodes and you could do replicas and things like that. And we want to allow users to configure control that type of thing down the line. But it is a fundamental trade off that you make in TurboPuffer right now, which means that your tail latency could be close to a second. There's lots of things you can do when you open a Q and a pane in Notion, they send a request to TurboPuffer to start warming the cache before the user has started typing in to get around these kinds of things. And we try to always improve the intelligence of the cache. Right. To make sure that it's evicting and keeping things in cache that are used. Those are the two fundamental trade offs. High ride latency means that like very complicated transactions and things like that is just. You're just not going to do that. But that's the fundamental trade off.

I buy that. So is that a primitive? That warming of the cache, is that a primitive you expose? So like if I have, we'll go back to music and say it's like, you know, post rock shoegaze or something that doesn't get pulled in to the cache very often. Do you. Does notion for example, just send off a query for post rock shoegaze or is that a primitive you expose? It's like, hey, warm up the cache somehow, some way.

Yeah, we just have an endpoint called hint cache worm. It's like m advice to occur.

I love that.

Yeah. And it's just if it's already in cache, it's free and if not, we charge you for one query.

That's great. I love that. So, yeah, totally affirmative. So S3 has dozens of competitors, some very popular, some that are just, just upstarts. But everybody claims S3 API compatibility that doesn't super matter for y', all because it sounds like the underlying part is what super matters for y'. All. Are there any other S3, are there any other object stores that offer the same kind of performance that you've looked at, that you're like, hey, maybe this is. This is an option that maybe has better performance or different performance?

TurboPuffer started on Google Cloud Storage Because Google Cloud Storage had compare and swap before S3. I don't know when they got it, but they had it in 2023 and S3 didn't. So we started on Google Cloud Storage and we were so adamant of not having a metadata layer. So adamant that in the beginning when we sold Notion is in AWS and we were in GCP and we bought like dark fiber interconnects with GCP because we needed to, we needed this compare and swap primitive so badly. And Notion was like, they, they understood. So. And unfortunately In Oregon, in US west, two or one depending on. Is it two in both? No, it's one and whatever. It doesn't matter. Those two data centers in Oregon are physically around 5 milliseconds apart. Like if you like have dark fiber, like in reality, should be less than that, but let's say 5 milliseconds. But they were showing up as 20 milliseconds going through Seattle. And so they're both like one, I think AWS is like in some suburb of Portland and the other one is like, like the GCP one is like out in the boonies. And they were going through an exchange in Seattle and It was like 20 milliseconds. So we bought like we bought through an exchange in Portland, setting all this up, like these were the lengths and we were tuning, we were tuning so much tcp because TCP by default, when you request data, you only get about 15 kilobytes of data. And then you have to wait for a round trip and then another round trip and then you get double like this TCP slow start. And so we were tuning that like on both ends and we were setting up proc. Like we did all this crazy stuff because Justine and I have been on call for a system that loses millions a minute for customers all around the world. And we did not want another stateless system. Sure, we'll buy all the state, state it stayed less stuff like all this, like Internet, all this stuff to make it fast enough. Now, of course we don't have to do this as much because we also run in AWS on S3. But to go back to your question, there's Google cloud storage, there's S3, right? There's Azure Blob Storage. We're starting to work with customers in Azure. I don't have too much to say about Azure Blob storage yet. But between GCS and S3, my main observations is the tail latency on S3 is higher, but the small object latency on S3 is lower. It's about 8 milliseconds. So if you're getting a very small object from S3, it's F than GCS. GCS for small objects is around 15 to 20 milliseconds. But the tail latency is better on GCS. Other than that, they're very comparable systems in our eyes.

So you've already gotten two of the other big clouds, so that would leave. I guess that leaves many. I know DigitalOcean has a compatible one. Cloudflare has a compatible one. I just talked to Tigris Data and that'll be coming out at this point that y' all are listening to this. That'll have been out for a week or so. So that's another one. But it sounds like you've got some of the big ones that your customers are looking for right now. How. How hard is it to maintain? Like, how different are those primitives? And do you just have, like, you know, adapters where in. Somewhere in the code? It's like, hey, I'm running on gcp. Do this one thing slightly differently.

Yeah, they just. We. We have our own client that we've written that works with all of our layers and it. I think you want to own your own client when you're doing this kind of stuff, there's a lot of things that you have to do, like signing the request. That's two days of my life that I never get back. And there's a lot of small, minor differences, but there's just if and else statements. But Azure Blob Storage is particularly annoying because they do not implement the S3 XML specs. So you have to do something completely different. And in terms of others, it will come from customer demand. If customers are asking for them, we will. Originally, TurboPuffer only worked on top of Cloudflare R2 and worked in Cloudflare Workers. That was like one of the. Yeah. The absolute first version of turbopuffer ran on Cloudflare Workers. There's probably still some WASM code sprinkled towards and around the turbopuffer code base. This was actually the original reason I had to write my own S3 client, because the other ones didn't work work for WASM, but I couldn't get it to be fast enough. So then I went this. This route.

So that's up. That's up to four major clouds that you have at least at one point written it for. That's. That's kind of wild. So where do we stand right now? So, turbopuffer, who do you serve? Like, what. What workload Use case, business type. Do you serve the most? And what do you see in the next six months a year for TurboPuffer? Where do you want to go?

We just want to get the whole world Puffin. I should be vaping on this call too. You know, just go look like you, you can, you can compile all of the world's knowledge into a couple terabytes of weights. And that model is going to have a very good idea about how to reason with the world. But in order for a model to reason with the outside world or the inside world of a corporation, it needs to search. That is the most important tool that it has in its arsenal. And TurboPuffer is going to index all of the private data in the world that wants to be connected to AI. So when we say that people want to connect data to AI, that's what we do. We also connect humans to data right at scales that were otherwise before, very, very difficult. So we see that in like so many different types of businesses, right? There's code, there's legal, there's hedge funds that use us, there's all kinds of businesses that have massive amounts of unstructured data and that unstructured data everyone is trying to get value out of. And that's what AI is so good at. And it's also what search is really, really good at sitting through. And so the way that I think about it is the turbopuffer is you give Turbo Puffer the haystack and we will give you the chunk of hay that the needle is in. But the LLM is very good at then using the needle and using that chunk of hay and getting value out of it.

That's a fantastic analogy, by the way. That's great. Okay, that makes sense to me. All right, so what do you see? Like product wise, give me a sense of where Turbo Puffer is at and any sense of scale you want, employees, anything public you want to share, like, give me a sense of the company.

Right now we're just shy of 20 people. We are very focused team of people who just love to build databases and people who want to, who love helping people build amazing things with those databases. And those are the kinds of people that I think would love working here and are working here now. We power more than a trillion vectors. I haven't heard of anything else than that. I'm sure the hyperscalers are at that scale. But to give you a sense of scale, the entire public Internet is in the hundreds of billion depending on how you do it. So this is extremely sizable Scale at peaks, we do more than 10 million vectors writes per second and we do tens of tens of tens of thousands of queries per second, which to me is not as impressive. Like you know, having worked on large MySQL clusters that do millions per second. So these are the kinds of numbers that we operate at. So this is real scale. Like it's like on some attributes larger than the scale that I was at, that I worked with at Shopify. And yeah, we have many, many, many customers that work with us and trust us.

Okay, so you're 2ish years old, 20 people doing huge scale. I want to talk about where you're going, but just on the employee thing, there are a lot of nerds listening to this. Are you actively looking for people and if so, what types?

We are always looking for people to join the team. We're looking for customer engineers, we're looking for sales, we're looking for database engineers, we're looking for people to work on the dashboard. We're looking for more or less every role that is interested in coming to build a database.

Okay, perfect. And I imagine probably Simon at TurboPuffer or something else. Like they can probably find you somewhere if they're clever enough. Right.

You should be able to find me. And I think you will have a higher probability of getting a great answer if you go to jobs on the website because then it doesn't land in my inbox, which is getting a little bit on, on like hard to stay on top of.

Fair enough. Okay, all right, so you all heard that if you're interested, go check that out. Now what, what's the next, what is the next six months or a year? You want to, you want to get the whole world puffin, if I may borrow the phrase on the product side. What does the next six months, a year look like? Are you just continuing to refine and like not stabilize in the fact that it's not stable, but refine and harden and like just keep going with where you're puff up, we'll say. Or is, are there more things on the horizon that you're like, hey, this is search adjacent or search compatible? Let's go ahead and like pull this in.

Look, we want to. Right now we are focused on building the best search engine and making that scale. So we work a lot on performance. We look a lot on our customers query plans and expand them. We are very focused on more full text search features. One of the, one of the people we just hired has been committing to Lucene for more than 10 years. And we're working on just adding more and more text features to the product. We are working on puffing up the dashboard so the, the. All the UX around using the product. Right now if you log into the TurboPuffer dashboard you will feel like this was vibe coded like by a 14 year old. And that's right. That's because this was sort of written the initial version by me over like a few months early on when I early on at Turbo Puffer. And then an amazing support engineer has worked on it since in between answering customers. But it's starting to get some love and I think people should get really excited about that. But it's really just we want to build a really good search engine and of course we want to expand with helping of many of the workflows around search. But we really are focused on creating an incredible product at the core before expanding with more auxiliary offerings around it.

Okay. Yep, that makes, that makes a lot of sense to me. This has been great. You're really good at explaining all of this highly technical stuff so well, well done you. As we, as we wrap here, tell the people, tell the people why they should consider turbopuffer and when they should consider Turbo Puffer. This is your like no holds barred, give us the pitch.

If you are searching data and you don't have that much data, then you should not be using TurboPuffer. You should just do whatever you have right there, put it in PGA Vector or whatever and go find PMF and then come, come find us when you have scale the. But to actually be serious here for a second and give some ballpark numbers. Any search engine is going to work in the millions. So you choose there based on one that you feel has the documentation and the guides and everything to get you what you need in the tens of million. We start to see people switch to us from some of the traditional storage architectures. Once you get into the hundreds of million or the billions of vectors or documents for full text search, there are very few things that work. We ran that at Shopify at that scale with elasticsearch and it took a very large team, like I think 20 plus even when I was there. And they're probably more people now to run these clusters. And it was very difficult to keep them alive and it was very difficult to use them for all the things that we wanted to use search for inside of Shopify at the time. And so I think that if you're, if you're dealing with operational pain or you're dealing with cost, holding back product or you're dealing with performance problems, then, like, Turbo Puffer should be an amazing solution. And if it's not, it's a bug.

Okay, that's a good pitch. I like the anti pitch at the beginning that, you know, you said it kind of tongue in cheek, but I like that. Like, if it's working, go find your customers. Keep using PG Vector when you run into a wall. Let's start puffing. Did I use that right? Let's start puffing. Is that good?

Killed it, Aaron.

Okay, that's perfect. Well, tell the people where they can find you. You where they can connect with you online.

They can find me. They can find me on X. Most of my posts are written while I'm running, which is a terrible habit, so it's a mixed quality, but X.com Syrupsen S I R U P S E N and then of course, turbopuffer.comx.com turbopuffer that's where it'll be good to find.

Okay, well, this has been a delight. Thank you for joining and thank you all for listening. If you are listening on audio, please go to YouTube and, you know, hit subscribe, like, whatever. If you're listening on YouTube and you're like, I kind of don't want to watch two guys talk. I'd rather listen to them. There's an RSS feed. You can find everything@databaseschool.com and please let me know. I always say this at the end to all of the dear listeners, if there's somebody you want on who is doing interesting things, who is not a suit, it's fine to have CEOs on, but they have to be technical. Let me know who you want to have on, who you want to hear nerd out for an hour or so. I would love to talk to more people. This has been an absolute delight. Simon, thank you for joining.