Qdrant x Dust: How Vector Search helps make work work better - Stan Polu | Vector Space Talk #010

Alright, so my man, I think people are going to want to know all about you. This is a conversation that we have had planned for a while. I'm excited to chat about what you have been up to. You've had quite the runaround when it comes to doing some really cool stuff. You spent a lot of time at Stripe in the early days, and I imagine you were doing lots of fun ML initiatives. And then you started researching on LLMs at OpenAI, and recently you are doing the entrepreneurial thing and following, uh, the trend of starting a company and getting really cool stuff out the door with AI. I think we should just start Yep. With background on yourself, what did I miss in that quick introduction?

Okay, sounds good. Yeah, perfect. No, you didn't miss too much. Maybe the only point is that starting the current company Dust with Gabriel, my co-founder, with whom we started a company together 12 years or maybe 14 years ago, I'm really bad with years, that eventually got acquired to Stripe. So that's how we joined Stripe, the both of us, pretty early. Stripe was 80 people when we joined. All the way to 2,500 people and got to meet with and to work with Greg Brockman there. And that's how I found my way to OpenAI after Stripe when I started interesting myself in research. At OpenAI, even if I'm not a trained researcher, I did research on, or faked doing research on large language models, reasoning capabilities, and in particular, large language models, mathematical reasoning capabilities. And And from there, 18 months ago, kind of decided to leave OpenAI with the motivation that is pretty simple. It's that basically the hypothesis is that those, and it was pre-ChatGPT, but basically those large language models, they're already extremely capable and yet they are completely underdeployed compared to the potential they have. And so while research remains a very active subject and it's going to be a tailwind for the whole ecosystem. There's probably a lot to be done at the product layer. And most of the locks between us and deploying that technology in the world is probably sitting at the product layer as it is sitting at the research layer. And so that's kind of the hypothesis behind DUST is to really try to explore at the product layer what it means to interface between models and humans, try to make them happier and augment them with superpowers in their daily jobs.

So you say product layer. Can you go into what you mean by that a little bit more?

Well, basically we have a motto at DUST, which is no GPU before PMF. And so the idea is that while it's extremely exciting to train models, it's extremely exciting to fine-tune and align models. There is a ton to be done. Above the model, not only to use them as best as possible, but also to really find the interaction interfaces that make sense for humans to leverage that technology. And so we basically don't train any models ourselves today. There's many reasons to that. The first one is as an early startup, it's a fascinating subject and fascinating exercise. As an early startup, it's actually a very big, investments to go into training models, because even if the costs are not necessarily big in terms of compute, it's still research and development and pretty hard research and development. It's basically research. We understand pre-training pretty well. We don't understand fine-tuning that well. And so we believe it's a better idea to really try to explore the product layer. The image I use generally is that training a model is very sexy and it's It's exciting, but really building— you're building a small rock that will get submerged by the waves of bigger models coming in the future. And iterating and positioning yourself at the interface between humans and those models at the product layer is more akin to building a surfboard that you'll be able to use to surf those same waves.

Ooh, I like that because I am a big surfer and I have a lot of fun doing it. Now tell me about, are you going after verticals? Are you going after different areas in a market, a certain subset of the market? How do you look at that?

Yeah, so we've made it. So basically, the idea is to look at the productivity within the enterprise. So we're first focusing on internal use by teams, internal teams of that technology. We're not at all going after external use. So backing products that embed AI or having our own projects maybe exposed through our users to actual end customers. So we're really focused on the internal use case. So the first thing you want to do is obviously, if you're interested in productivity within enterprise, you definitely want to have the enterprise data, right? Because otherwise there's a ton that can be done with ChatGPT as an example, but there is so much more that can be done when you have context on the data that comes from the company you're in. And yeah, that's pretty much kind of the use case we're focusing on. And we made, we're making a bet, which is a crazy bet to answer your question, that there's actually value in being quite horizontal for now. So that comes with a lot of risks because an horizontal product is hard to read and hard to it's hard to figure out how to use it. But at the same time, the reality is that when you are somebody working in a team, even if you spend a lot of time on one particular application, let's say Salesforce for sales or GitHub for engineers or Intercom for customer support, the reality of most of your workflows do involve many SaaS, meaning that you spend a lot of time in Salesforce, but you also spend a lot of time in Slack and Notion maybe. We all spend as engineers a lot of time in GitHub, but we also use Notion and Slack a ton or Google Drive or whatnot.

Jira.

And so, or Jira, the, the, the, the good old Jira. Beloved Jira.

Everybody loves spending time in Jira. Yeah.

And so having basically following our users where they are requires us to have access to those different SaaS, which, which requires us to be somewhat horizontal. We've seen, we had a bunch of signals that kind of confirms that position. And yet, we're still very conscious that it's a risky position. As an example, when we are benchmarked against other solutions that are purely verticalized, there is many instances where we actually do a better job because we have access to all the data that matters within the company.

Now there is something very difficult when you have access to all of the data, and that is the data leakage issue and the data access, right? How are you trying to conquer that hard problem?

Yeah, so we're basically focusing— so to continue answering your questions through that other question, we— I think we're focusing on, on tech companies that are less than 1,000 people. And if you think about most recent tech companies less than 1,000 people, there's been a wave of openness within companies in terms of data access, meaning that it's becoming rare to see people actually relying on complex ACL for the internal data. You basically have, generally have silos. You have the exec silo with remuneration and ladders and whatnot. And this one is definitely not the kind of data we're touching. And then for the rest, you generally have a lot of data that is accessible by every employee within your company. So that's not a perfect answer, but that's really kind of the approach we're taking today. We give a lot of control on which data comes into Dust, but once it's into Dust, and that control is pretty granular, meaning that you can select specific Slack channels, so you can select specific Notion pages, or you can select specific Google Drive subfolders. But once you decide to put it in Dust, every Dust user has access to this. And so we're really taking the silo vision of the granular ACL story. Obviously, if we were to go higher enterprise, that would become a very big issue because I think larger are the enterprise, the more they rely on complex ACLs. Mhm.

And I have to ask about your history with Stripe.

Yep.

Have you been focusing on specific financial pieces to this? Like, uh, first thing that comes to mind is what about all those e-commerce companies that are living and breathing with Stripe? Feels like they've got all kinds of use cases that they could leverage AI for, whether it is their supply chain or just getting better numbers or getting answers that they have across all this disparate data. Have you looked at that at all? Is that informing any of your decisions that you're making these days?

No, not quite. Not really. At Scribe, I mean, when we joined, it was very early. It was the quintessential curl/v1/chargers dash dash number 42, 42, 42. And that's pretty much what Stripe was almost. I'm exaggerating, but not too much. So what I've been focusing at Stripe was really driven by my and our perspective as European founders joining a quite US-centric company, which is, no, there is not credit card all over the world. Yes, there is also payment methods. And so most of my time spent at Stripe was spent on trying to expand the API to not a couple US payment methods, but a variety of worldwide payment methods. So that requires kind of a change of paradigm from an API design. And that's where I spent most of my cycles while I was at Stripe.

Okay. The next question that I had is you talked about how benchmarking with the horizontal solution surprisingly has been more effective in certain use cases. I'm guessing that's why you got a little bit of love for Qdrant and what we're doing up here.

Yep. Uh, so I mean, it's really— I think it's, uh, there's, there's— the benchmarking was really about quality of models answers in the context of retrieval augmented generation. So it's not as much as performance, but obviously performance matters. And that's why we love using quadrants. But I think the main idea of what I mentioned is that it's interesting because the mode today, the retrieval is noisy because the embedders are not perfect, which is an interesting point. Sorry, I'm double-clicking, but I'll come back. But it's an interesting point.

Love them. Let's go.

The embedders are really not perfect. So that's interesting as when Quadrant released kind of optimization for storage of vectors, they come with obviously warnings that you may have a loss of precision because of the compression, et cetera, et cetera. And that's funny, like in all kind of retrieval augmented generation worlds, it really doesn't matter. We take all the performance we can because the loss of precision coming from compression of those vectors at the vector DB level are completely negligible compared to the all-around fuckness of the embedders in terms of capability to correctly embed text because they're extremely powerful, but they're far from being perfect. And so that's an interesting thing where you can really go as far as you want in terms of performance because your error is dominated completely by the quality of your embeddings. And so, yeah, going back up, I think what's interesting is that the retrieval is noisy because of the, mostly because of the embedders and the models are not perfect. And so the reality is that more data in a RAG context is not necessarily better data because the retrievals become noisy, the model kind of gets confused and they start hallucinating stuff, et cetera. And so the right trade-off is that you want access to as much data as possible, but you want to give the ability to our users to select very narrowly the data required for a given task. And so that's kind of what our product does, is the ability to create assistants that are specialized to a given task. And most of the specification of an assistant is obviously a prompt, but also saying, oh, I'm working on helping sales find interesting next leads. And you really want to narrow the data exactly where that information lies. And that's where there we're really relying hard on Qdrant as well. So the kind of indexing capabilities on top of the vector search, where whenever we insert the documents, we kind of try to insert an array of parents that reproduces the hierarchy of whatever that document is coming from, which lets us create a very nice user experience where when you create an assistant, you can say, oh, I'm going down two levels within Notion and I select that page and all of those children will come together. And that's just one string in all specification because then we rely on those parents that have been injected in Quadrant. And then the Quadrant search really works well with a simple query like this thing has to be in parents and you filter by that.

And it feels like there's two levels to the evaluation that you can be doing with RAGs. One is the stuff you're retrieving and evaluating the retrieval, and then the other is the output that you're giving to the end user. Yeah. How are you attacking both of those evaluation questions?

Yep. So the, uh, the truth, uh, in whole transparency is that we don't, uh, we're just too early.

Well, I'm glad you're honest with us at least. Shit.

This is great. We should, we should, but the reality is that we have so many other product priorities that it's, it's more, it's, it's, it's, I mean, I think evaluating the quality of retrievals, evaluating the quality of retrieval augmented generation, um, good sense, but good sense is hard to define because good sense with 3 years doing research. In that domain is probably better sense, better good sense than good sense with no clue on the domain. But basically with good sense, I think you can get very far and then you'll be optimizing at the margin. And the reality is that if you get far enough with good sense and that everything seems to work reasonably well, then your priority is not necessarily on pushing 5% performance, whatever is the metric, but more like I have a million other product questions to solve. That is the kind of 10 people answer to your question. And as we grow, we'll probably make a priority, of course, of benchmarking that better. In terms of benchmarking that better, extremely interesting question as well, because the embedding benchmarks are what they are. And I think they are not necessarily always a good representation of the use case you'll have in your products. And so that's something you want to be cautious of. And it's very hard to, it's extremely, I mean, it's quite hard to benchmark your use case. The kind of solutions you have and the ones that seems more plausible without spending like full years on that is probably to evaluate, I mean, evaluate the retrieval with another model, right? It's like you take 5 different embedding models, record a bunch of questions that comes from your product, you use your product data and you run those retrievals against those 5 different Sambetas and then you ask GPT-4 to rate. That would be something that seems sensible and probably will get you another step forward and is not perfect, but it's probably really strong enough to go quite far. And then the second question is evaluating the end-to-end pipeline, which includes both the retrieval and the generation. And there, there it's a, to be honest, again, it's a non-question today because GPT-4 is just so much above all the models that there's no point evaluating them. If you accept using GPT-4, just use GPT-4. If you want to use open source models, then the question is more important. But if you are okay with using GPT-4 for many reasons, then there is no questions at this stage.

So my next question there, because sounds like you got a little bit of a French accent, uh, guessing you're somewhere in Europe. Are you in France, or are you—

Oh yes, we're based, we're based in France, and the Bloom team from, from Paris.

So I was wondering if you were going to lean more towards the history of you working at OpenAI or the the fraternité from your French group and go for your amis in Mistral.

And I mean, we are absolute BFF with Mistral. The fun story is that I knew, I mean, Guillaume Lampé is a friend because we were working on exactly the same subjects while I was at OpenAI and he was at Meta. So we were basically frenemies. We were competing against the same metrics and same goals, but grew a friendship out of that. We are, I mean, our platform is quite model agnostic. So we have, we support Mistral there. Then we do decide to set the defaults for our users. And we obviously set the defaults to GPT-4 today. I think this question of where, today there's no question, but when the time comes where open source or non-open source, it's, it's not the question, but where other models kind of start catching up with GPT-4, that's going to be an interesting product question. And hopefully Mistral will get there. I think that's definitely their goal to be within reach of GPT-4 this year. And so that's going to be extremely exciting.

Yeah. So then you mentioned how you have a lot of other product considerations that you're looking at before you even think about evaluation. What are some of the other considerations?

Yeah, so the main, as I mentioned a bit, one of the main, the main hypothesis is we're going to do company productivity, you got team productivity, we need the company data. That was kind of hypothesis number zero. It's not even an hypothesis, almost an axiom. And then the second hypothesis, our first product was a conversational assistant like ChatGPT that is general and has access to everything. And realized that didn't work quite well enough on a bunch of use cases. It was kind of good on some use cases, but not great on many others. And so that's where we made that first strong product hypothesis, which is, so we want to have many assistants, not one assistant, but many assistants targeted to specific tasks. And that's what we've been exploring since the end of the summer. And that hypothesis has been very strongly confirmed with our users. And so an example of issue that we have is, Obviously you want to activate your product. So you want to make sure that people are creating assistants. So one thing that is much more important than the quality of RAG is the ability of users to create personal assistants. Before it was only workspace assistants. And so only the admin or the builder could build it. And now we've basically, as an example, worked on having anybody can create the assistant. The assistant is scoped to themselves. They can publish it afterwards, et cetera. That's the kind of product questions that are To be honest, more important than write quality, at least for us.

Um, sorry, real quick, publish it for a greater user base or publish it for the internal company to be able to—

yeah, within the workspace.

Okay, it's not like, oh, I could publish this for—

yeah, no, no, no. Yeah, okay, we're not going there yet, and, uh, and there's plenty, plenty to do. Internally to each workspace before going there. Though it's an interesting case because that's basically another big problem is you have an horizontal platform, you can create an assistant, you're not an expert and you're like, okay, what should I do? And so that's the kind of white blank page issue. And so there having templates, inspiration, you can sit that within workspace, but you also want to have solutions for the new workspace that gets created. And maybe a marketplace is a good idea of having templates templates, etc., are also product questions that are much more important than the RAG performance. And finally, one of our, I mean, the users where Dust works really well, one example is Alain in France. There are 600 and Dust is running there pretty healthily and they've created more than 200 assistants. And so another big product question is like when you get traction within a company, people start getting flooded with assistants. And so how do they discover them? How do they know which one to use, et cetera? So that's kind of the kind of many examples of product questions that are very first order compared to other things.

Because out of these 200 assistants, are you seeing a lot of people creating the same assistants?

That's a good question. So far it's been kind of driven by somebody internally that was responsible for trying to push gen AI within the company. And so I think there's not that much redundancy, which is interesting. But I think there's a long tail of stuff that are mostly explorations. But from our perspective, it's very hard to distinguish the two. Obviously, usage is a very strong signal. But yeah, displaying assistance by usage, pushing the right assistant to the right user, this problem seems completely trivial compared to building an LLM, obviously. But it's still, when you are at the product layer, requires a ton of work. And as a startup, that's where a lot of our resources go. And I think that's the right thing to do.

Yeah, I wonder if, and you probably have thought about this, but if it's almost like you can tag it with this product or this assistant is in beta or alpha, or this is in production, like you can trust that this one is stable, that kind of thing.

Yeah, so we have the concept of shared assistant and the concept of workspace assistant. The workspace assistant are like the admin vetted the assistants and it's kind of pushed to everyone by default. And then the workspace, the published assistant is like there's a gallery of assistants that you can visit and there the strongest signal is probably the usage metric, right? Yeah.

Yeah. So when you're Talking about assistants, just so that I'm clear, it's not autonomous agents, is it?

No, yeah, so it's a great question. We are really focusing on the one step, trying to solve very nicely the one step thing. I have one granular task to achieve and I can get accelerated on that task and maybe save a few minutes or maybe save a few tens of minutes. On one specific thing, because the agentic version of that is obviously the future. But the reality is that current models, even GPT-4, are not that great at kind of chaining decisions of tool use in a way that is sustainable beyond the demo effects. And so while we are very hopeful for the future, it's not our core focus because I think there's a lot of risk that it creates more deception than anything else. But it's obviously something that we are targeting in the future as models get better. Yeah.

And you don't want to burn people by making them think something's possible and then they go and check up on it and they leave it in the agent's hands and then next thing they know they're getting fired because they don't actually do the work that they said they were going to do.

Yeah. One thing that we don't do today is, well, we have kind of a different ways to bring data into the assistant before it creates the generation. And we're expanding that. There's one of the main use cases, the one based on Qdrant, which is the kind of retrieval one. We also have kind of a workflow system where you can create an app, an LLM app, that's where you can make multiple calls to a model. You can call external APIs and such. And another thing we're digging into are structured data use case, which this time doesn't use Qdrant. Which the idea is that semantic search is great, but it's really atrociously bad for quantitative questions. Basically, the typical use case is you have a big CSV somewhere and it gets chunked and then you do retrieval and you get kind of a disordered partial chunks of that. And on top of that, the models are really bad at counting stuff. And so you really get bullshit.

You know better than anybody, huh?

Yeah, exactly.

Exactly.

And so you garbage in, garbage out, basically. And so, and so we're digging, we're looking into being able, whenever the data is structured, to actually store it in a structured way. And as needed, just in time, generate an in-memory SQL database. So that the model can generate a SQL query to that data and get kind of a SQL answer. And as a consequence, hopefully be able to answer cognitive questions better. And finally, obviously the next step also is as we integrated with those platforms, Notion, Google Drive, Slack, et cetera, basically there's some actions that we can take there. We're not gonna take the actions, but I think it's interesting to have the model prepare an action. Meaning that here is the email I prepared, send it or iterate with me on it. Or here is the Slack message I prepared. Or here is the edits to the Notion doc that I prepared. This is still not agentic. It's closer to taking action, but we definitely want to keep the human in the loop. But obviously some stuff that are on our roadmap. And another thing that we don't support is, which is one type of action would be the first we would be working on is obviously code interpretation, which is, I think it's One of the things that all users ask because they use it on ChatGPT. And so we'll be looking into that as well.

What made you choose Qdrant?

So the decision was made, if I remember correctly, something like February or March last year. And so the alternatives I looked into were Pinecone, Wavy8, some ClickOwl because Chroma was using ClickOwl at the time, but Chroma was 2,000 lines of code at the time as well. And so I was like, oh, Chroma. So we're part of AI grant and Chroma is as an example also part of the AI grant. So I was like, oh, well, let's look at Chroma. And however, what I'm describing is last year, but they were very early. And so it was definitely not something that seemed like to make sense for us. So at the end, it was between Pinecone, Wavev8, and Quadrants. Wavev8, you look at the doc, you're like, yeah, no, not possible. And then finally, it's Quadrants and Pinecone. And I think we really appreciated the, obviously, the open source nature of Quadrants. From playing with it, the very strong performance, the fact that it's written in Rust, the sanity of the documentation. And basically the feeling that because it's an open source, and we're using the hosted Quadrant Cloud solution, but it's not a question of paying or not paying. It's more a question of being able to feel like you have more control. And at the time, I think it was the moment where Pinecone had their massive fuck-up where they erased a gazillion database from their users. And so we've been on Quadrants and I think it's been a two-step process. Really, it's very smooth to start, but also Quadrants at this stage comes with a lot of knobs to turn. And so as you start scaling, you at some point reach a point where you need to start tweaking the knobs, which I think is great because you— the knobs, there's a lot of knobs, so they are hard to understand, but once you understand them, you see the power of them. And the Quadrant team has been excellent there supporting us. And so I think we've reached that first level of scale where you have to tweak the knobs, and we've reached the second level of scale where we have to have multiple nodes. But so far it's been extremely smooth and I think we've been able to do with Squadron some stuff that's really all possible only because of the very good performance of the database. As an example, we're not using your clustered setup. We have a number of independent nodes. And as we scale, we kind of reshuffle which users go on which nodes as we need. Trying to keep our largest users and most paying users on very well-identified nodes. We have a kind of a garbage node for all the free users. And as an example, migrating even a very big collection from one node, one capability that we build is say, oh, I have that collection over there. It's pretty big. I'm going to initiate on another node. I'm going to set up shadow writing on both and I'm going to migrate live the data. And that has been incredibly easy to do with Squadron because scrolling is fast, writing is fucking fast. And so even a pretty large collection, you can migrate it in a minute. And so it becomes really within the realm of being able to administrate your cluster with that in mind, which I think would have probably not been possible with a different system.

So it feels like when you are helping companies build out their assistance, are you going in there and giving them ideas on what they can do?

Yeah, we had, we added at a stage where obviously we have to do that because, uh, I think the product basically has starts to have strong legs, but I think this is still very early. And so there's still a lot to do on activation as an example. And so we are in a mode today where we do what doesn't scale basically. And we do spend some time with companies, obviously, because there's no way around that. But what we've seen also is that the users where it works the best and being on Dust or anything else that is relative to having people adopt GenAI within the company is companies where they actually allocate resources to the problem, meaning that the companies where it works best are the companies where there's some, somebody, their role is really to go around the company, find use cases, support the teams, etc. And in the case of companies using Dust, this is kind of type of interface that is perfect for us because we provide them full support and we help them build whatever they think is valuable for their team.

Are you also having to be the bearer of bad news and tell them like, yeah, I know you saw that demo on Twitter, but that is not actually possible or reliably possible?

Yeah, that's an interesting question. That's a good question. Not that much because I think one of the big learning is that you take any company, even a pretty techie company, pretty young company, and the reality is that most of the people, they're not necessarily in the ecosystem, they just want shit done. And so they're really glad to have some shit being done by a computer, but they don't really necessarily say, oh, I want the latest, shiniest thingy that I saw on Twitter. So we've been safe from that so far.

Excellent. Well, man, this has been incredible. I really appreciate you coming on here and doing this.

Yeah.

Thanks so much. And if anyone wants to check out Dust, I encourage that they do. It's dust. It's a bit of an interesting website. What is it? Dust..tt..tt. That's it. That's what I was missing. Dust..tt. There you go. So if anybody wants to look into it, I encourage them to. And thanks so much for coming on here.

Yeah. And Quadrants is the shit.

There we go. Awesome, dude. Well, this has been great.

Yeah, thanks. Me too. Have a good one.