Glean Founders Talk AI-Powered Enterprise Search on NVIDIA Podcast - Ep. 190

Foreign.

Welcome to the Nvidia AI Podcast. I'm your host, Noah Kravitz. Large language models and generative AI are all the rage as 2023 gets rolling. Google, Microsoft and other tech giants have been busy announcing AI powered versions of their search engines and other next gen products, leveraging the power of LLMs. But some say the opportunity for surge is bigger inside of companies than outside of them. And that's where today's guests come in. Didi Das and Eddie Zhou are founding engineers at Glean, a Silicon Valley startup building better tools to help people find what they need at work. Last year, Glean raised a Series C financing round, valuing the company at $1 billion capital the company is using to create better enterprise search with cutting edge AI. Didi and Eddie are here to talk about the challenges and opportunity in bringing LLMs into the enterprise. So let's get right to it, gentlemen, welcome and thank you for taking the time to join the Nvidia AI podcast.

Thanks, Noah. Pleasure to be here.

So I'll let you decide who to take this. But to start off, can I ask you to just describe Glean for the audience?

Yeah, absolutely. So, you know, enterprise search is been a problem, Noah, since the 90s, companies have had problems finding information at work for a very, very long time. And Glean is, I think, is finally doing that right. So we actually index everything that you might have access to in your company and allow you to search through everything at one go, including Slack, you know, Dropbox your mail, what have you, and find exactly what you're looking for, wherever you need it at any given time in under a second. So it's really bringing that Google experience that we all know and love to the enterprise, to the organization, for every knowledge worker to use.

So what was the impetus for getting involved with Glean? And we can take that back or we can a little later in the podcast, go into your own backgrounds. But how did you start getting involved? Working with LLMs, working with generative AI and then landing at Glean.

So I, you know, I worked on the first deep learning model we launched in Google Web search back in 2015. And just seeing how that technology made its way throughout Google, working on the brain team there until it permeated so many different aspects of Google products was just amazing. And, you know, almost four years ago now, when I met up with Arvind and heard his, his pitch about bringing a lot of this technology to knowledge workers who really had to deal with not having anything of the sort, it was just, frankly, inspiring and so that was kind of how I came to glean. And then in recent years we've been building a lot of that same technology into Glean. And in the past year seeing all these awesome breakthroughs on the generative side has really got our mind spinning and us thinking about how we can bring that as well into gleam.

Eddie, I'm wondering, you were at Google and it made me think. I've worked at companies mostly doing writing where you know, we've dog fooded, right. We've used our own products internally. But when I've been doing research for a piece of marketing, content marketing I've been working on, I'll use Google Advanced search all the time to search within the site domain of the company I was working for. So to find backlinks or, you know, previous posts on oh, you're a real power user. No, I pretend to be one anyway, but that, you know, the big difference as I'm listening to you talk about glean and was prepping for this conversation is that you know, that can search my company's website. So in this case, you know, I could, I could tell it to search Nvidia.com, but we're not searching any of the internal repositories, the slack, the email documents, whatever system we're using for you know, an internal social network type of, of messaging, that kind of thing.

When you were at Google, were those.

Kind of advanced search functionality features baked into internal products that you used? Or were you like the rest of us where, you know, as you described knowledge workers without the power to search their companies on knowledge, to put it that way?

Yeah, that's, that's a great question. And you know, Didi was at Google as well, so he'll know this. But you know, there was an internal search tool. I think over the course of my time there, it did get a lot better. People, you know, kind of took it for granted. A lot of folks who left Google, went out into the world, worked at smaller companies or other companies, found, oh wow, I don't have this tool anymore. And it really impacted their productivity. I think a good amount of larger tech companies tend to end up rolling their own internal search to solve for this. But you know, over the past few years what we've seen is like the fragmentation and all these SaaS apps makes it really hard to actually have a good solution that scales and that's what we've been trying to build.

So Didi, you were at Google as well and then did both of you move from Google directly to Glean or were you at other places in between?

No, I moved directly from Google to Glean. I've sort of always been passionate about search as a problem. That was the only team on Google I wanted to work on. And when I heard that Glean was building enterprise search, it was, it was very obvious because the tool that he talked about, moma, was even that was probably the best in the industry, but it doesn't even apply to most other companies. And the quality there could be a lot better too. So I was immediately attracted to this problem and came right after Google.

And so what LLMs offer for enterprise search, that just hasn't been possible previously? Or what are some of the things that using an LLM can unlock for an enterprise?

Yeah, that's a great question, Noah. I think, as folks have seen the ChatGPT demos played with it themselves, the question inevitably arises, hey, how do I use this at work? How could that look in a product? And for us, I think there's a really, really big space of opportunities here. These models are highly fluent and coherent and capable of very strong synthesis abilities. And so all of those things really neatly fit into our company's core vision of helping folks find information, synthesize information, and do things with that information. Right. So you can imagine the kind of maybe first application that we're working on is a way to augment a search results page and give kind of synthesis a direct answer, using the results that are there anchoring on them to give a direct answer to the question that someone might ask. But that's really just the tip of the iceberg in terms of what we can do with this technology. That being said, there's many challenges which we're happy to talk about as well, that we're trying to solve to actually get that technology in the hands of everyone.

Well, since you brought it up, I've got a couple in mind. But would you speak to some of those challenges and, if applicable, how you're tackling them?

Yeah, sure. So I think that the primary challenge that we see and the kind of discourse around ChatGPT has been basically this, this kind of conflation between coherence and knowledge.

Right?

These models are trained over this massive amount of data, and so they, they have built an implicit world model inside them, and also their generated output is highly coherent and fluent. So then you can kind of treat them as this world search engine, right? You can ask it questions and it's finding stuff from inside. But the tricky part in where it comes to bringing this to the enterprise, to knowledge Workers to internal knowledge is that's all knowledge behind, you know, behind walls. These models have never seen them before. And so if you don't do the appropriate work involved in calibrating those models or understanding that challenge, if you don't connect that model to the right knowledge, you know you're going to end up with stunningly inaccurate results, confidently inaccurate results, and altogether just not deliver a good experience.

I like that phrase confidently inaccurate. That describes some of my own, my own dealings with ChatGPT as I'm sure anyone who's tried it can relate to. So when you work with a new client, do you train a model on their internal data? Is it a separate model? Do they kind of get baked together with, you know, the LLM LLMs that you're using? How does that work?

Yeah, that's a great question. Right now, today we do train a language model per customer and you know, we can talk a bit more about security aspects down the road, but you know, that's always front of mind for us. We don't let this model, you know, come out of that customer deployment. And so the aim here is really to, to transfer world knowledge in, calibrate that model, train it on the specific company's documents and you end up compressing, you know, that company's language into that model. And that can drive a lot of really cool things in search.

Yeah, it brings to mind for me, I mentioned, you know, I do a lot of work outside the podcast writing and I've been playing with some of the various AI writing assistant tools that are out there and there are a few that offer the possibility of learning your house style. And that really makes me think about, you know, everything from can I clone myself and written form to, you know, that, that reinforcing of every time I use it, it understands a little bit better what I'm looking for, how to respond to me, how I query it, those sorts of things. Is that something that Glean is doing or looking into? Is that something that customers are interested in? What are some of the, I don't know, some of the interesting specifics when it comes to kind of a custom trained LLM and this whole technology on a per customer basis.

Didi, you know what, you take this one.

Thanks, Eddie. I just wanted to make the point that for the LLMs in glean use case, we haven't specifically looked into writing assistance so much. It's, it's really doesn't align with our real core vision, but there's so many other ways.

Absolutely. I was just going to Say I have sort of a good slash bad habit of relating everything back to myself to make sense of it. So yes, I didn't mean to imply that, that that was the business you guys are in.

No, but there's, there's so many other ways you could see that elements could appear in the enterprise and be really valuable. So one of the things that we do and Eddie would, would have so much more context on this is we have a Slack bot. So when you are in a Slack channel and you ask a question, you might be asking it to your whole company. And an LLM can behind the scenes say, hey, we understand that you're asking about this project. We understand that we have data from this project from a week ago and then just say, hey, the status of this is this and nobody else has to in your company jump in or you don't have to wait for anybody to come in and answer that question anymore.

That's great. I'm sold immediately. As soon as you phrase that, I thought, oh, that makes a lot of sense. How does it glean interfaces you just mentioned through a Slack bot? What are some of the other ways that workers in a company could leverage Glean? How does the interface, excuse me, work?

Well, there's many other interfaces. Slack Bot is just one of our many entry points. We have interfaces from a homepage, a new tab page where users can go to and have a search bar for their entire company. And we even embed ourselves natively within every other search bar you might use in your company. So if you're on Google Drive and you're trying to look for something, a lot of people tell us Google Drive search is just not that great. So when they try searching on Google Drive, we would replace that entire experience with clean and you'd be able to find your Google Drive documents that way.

That's fantastic. Eddie brought up security, so maybe let's dive into that. How do you handle security? I would assume that the Glean data is in the cloud somewhere. It's not an on prem thing that's set up on a client site. But how does that work?

Yeah, that's a great question. You know, we've built this company and a product with security first from day one. And in terms of where the data is stored, we work with customers in a couple of different configurations. But we've managed to seal deals with security companies themselves, which is always a, a great sign that, you know, we're doing something right. So they've built that trust with us in terms of how we lay things Out. I think one, one dimension of security that's really important to call out is security within the company itself in terms of access, right? So, you know, solving the access problem for search is technically challenging on its own, but that's what we've done. We, we've built a system that within each data source we connect to, respects the permissions and the identity of the user who's issuing that query so that they can only ever see documents that they actually have access to. Right? And that's pretty, pretty clear. It's a technical challenge, but it's a clear thing. Now what LLMs brings to that is a lot more complication, right, because you can train a model appropriately on data that is only accessible to certain people, but once you start talking about generated output, that's when the line gets a little bit blurry. I can very clearly tell you, hey Noah, you don't have access to this document.

Right?

But can I tell you, hey, Noah, you don't have access to these three words in sequence, right? Like, that's the line that we're talking about now with figuring how to make kind of permission sensitive generated output work within a product. So LLMs definitely bring a new facet of challenge to this permissions model that we worked so hard to build.

One of the most mentioned issues with LLMs kind of in the mainstream, and we're really talking about a matter of months since, you know, the masses have had access to ChatGPT and some of the other LLMs and apps built on top of them. But one of the big issues is what people call hallucinations. So incorrect results, you know, confidently inaccurate answers to questions, that sort of thing. So with a service like ChatGPT or some of the others built on OpenAI's LLMs, you know, there's a caveat that it's not an Internet search, it's not a live search, and it's trained on data that goes up to, you know, 2019 or whatever it is. And so, you know, expect inaccuracies. Are hallucinations more or less or kind of the same problem or, you know, same sort of intensity and volume of problem in an enterprise as opposed to, you know, in an LLM out in the wild. And then kind of as a second related question, how do you ensure that, you know, at a big enterprise company, there's, there's data being created by the minute, new documents, new chats, new slacks, everything. So how do you keep up with that? How do you ensure that an enterprise customer using Glean, has access to you Know, as close to real time internal data as possible.

Yeah, I can, I can say a little bit about the hallucination as it relates to the Enterprise and maybe toss it over to Didi to talk about the freshness aspect. But on the hallucination side, I think in the Enterprise, you know, it's a very similar problem. The stakes are a little different as they are for any product.

Right.

You know, I'll say even today, people without generated output can find, you know, a slack message of someone saying the wrong answer to a question. Right, sure. Now they think that as well. And so misinformation, so to speak, will spread. Right. And that's obviously the case in the public web, in the public search domain as well. And generated kind of output can kind of just further exacerbate that. Right. So the stakes are different in the sense of, you know, within a company, generally speaking, you can rely on not having bad actors and you're all trying to build the same thing, move the same direction, and there's a little bit of chaos there, but in general you're aligned. Right. So it's a little bit simpler of a problem. But, but that being said, the hallucinations are still very real. And we've been spending the past few months kind of working within this very large body of literature in the research community that seeks to figure out how do we anchor generated output against a real knowledge base. How do we reconcile factuality, the real world, with the inherent kind of token driven nature of generating output from these LLMs. And there's lots of really, really cool work happening done by all these awesome researchers that we're just trying to build on top of and figure out how do we reconcile, hey, this is what the model said. Can we use what we know maybe of the enterprise, of the corpus as it exists to confirm that, to reconcile that Right. Before we just blindly send it off to the user? Because it's really important to kind of keep a handle on what that generation looks like and be transparent to the user with what you're giving them.

So if you tack onto the freshness side of what Eddie said, it's very related to hallucinations too. I mean, even today in search, you might ask a question which is factually correct, but temporarily it's not. So it's correct as of maybe five minutes ago, but it's no longer correct. And that kind of brings us to the open world nature of search, where you can't help, you can't try to even answer a question that you don't know the answer to and the temporal nature of search is exacerbated in the enterprise where like you said, like Slack, messages are being sent on a daily basis. Information is changing in the order of minutes and not days. So you know, just like Eddie said, retrieval augmented solutions do add a lot of value here and help anchor on factual things. But even then, LLMs as we know them today are really hard to keep fresh. So you need to use some of these techniques to ensure the final thing that you're saying is accurate.

My guests today are Didi Das and Eddie Zhou. Didi and Eddie are founding engineers at Glean, a Silicon Valley startup that is leveraging AI and large language models amongst other technology, I'm sure to build better search, better search experiences for the enterprise so that knowledge workers in particular can, can get easier, faster, better answers to their questions in a work context. I'm wondering kind of on a broader scale. There's obviously been a lot of talk over the past year or two in particular, but for way back about the impact that AI is having now is going to have on the workplace. So everything from fears that the robots are going to take our jobs to this notion that the age of AI will usher in the age of abundance because it will become faster and faster and easier and easier for us to let the machines take care of all the automated parts of the work and so we can build more of what we need faster and we can, we humans can move on to more interesting kind of creative, high level thinking pursuits. Does Glean have a vision for the future of work per se, or if that's not the right way to ask it, what, what do you guys think about it? What do you see? You know, the, the chatbot answer you mentioned really lit a light bulb for me a couple minutes ago when I thought, right, if there's something I don't know and I can't find it and I ask on, on Slack or what have you, it's a matter of taking away one of my colleagues bandwidth to answer my question and also having to wait for them to answer it. Whereas a Glean powered bot that can go fetch the answer and you know, report back in a, in a coherent, easy to digest way. Bam, there you go. It's a win, win all around. What are some of the other implications that you guys that Glean see for the future of AI supercharged work?

I mean, I think, you know, Noah, you answered your own question quite well there as it relates to our vision. Just to kind of emphasize some of the things you mentioned there I think we're firmly in the boat of taking the work that people don't want to do that actually wears them down, that ends up making them more disengaged at work. We want to take that work and figure out how we can leverage technology, leverage AI to make it happen. Right. And the kind of question answering bot or chatbot you mentioned is one variant of that. But there's a lot of things that people kind of reconcile that they have to do in their day to day that they think, okay, I'm the only one who can do this because I'm the one that has this knowledge. You know, I have this context within the enterprise and there's lots of opportunities there for us to help kind of magically insert the right technology, help those people in the moment and kind of remove the tedious work that they don't actually want to do. You know, and search really still actually fundamentally fits into that framework. Right. Finding information, that the cost of not finding information is, is incredibly high. You know, this, this number has floated around a bunch. But you know, a recent Forrester report says knowledge workers spend 12 hours a week looking for information. Right. And no one wants to be spending that time.

Right.

No one wants to actually be asking their coworkers a bunch of stuff, going from tool to tool. And what they want to do is actually do interesting creative work that's fulfilling that, that's actually in line with, you know, their skill set and what they want to do. And so that's really firmly how we think about the optimism within, how AI can help empower knowledge workers. Right. Not replace them, but rather augment what they can do.

Yeah.

And I want to add, just to tack on to Eddie's point, that's very well said. And I know there's a lot of doom and gloom about hey, will AI take our jobs? But every innovation for a while, when you think about machines or computers, people thought that about those things too. And I think what they miss is it doesn't actually take your job. It just frees you to do things that you're better suited to do. You don't want to spend all your time hand knitting clothes. You want to do some higher level work than that. And it makes humans as a whole much more enabled to do what they need to do in the world and makes global productivity go up if they don't have to waste time looking for information. I think at the workplace that's where glean comes in.

One of the things on the consumer side that really, you know, just, just in the recent months has has started to become something to keep an eye on. Are the cost models for accessing LLMs? ChatGPT recently introduced a premium tier. I think it's $20 a month and I mentioned before my own interest in writing tools and they have different models based on a monthly fee, sometimes with, you know, sort of a monthly and then you have a usage fee. So depending on how many words you have a generate each month, that sort of thing. Is cost a big concern when it comes to using LLMs in the enterprise? And if so, how do we keep the cost reasonable so that the individual companies. But then Didi, to your point, kind of all of humanity can benefit from this sort of increased production but it's easier and I don't have to do the things I don't like doing so I can focus on the more interesting work. It has cost figure into all of this.

Yeah, well it really depends on how you use LLMs that the cost structure really varies. Now if you're generating, if you're doing a writing tool and you're selling to customers, then you have to charge because you are paying for this and there's no money coming in. In some sense we're in a privileged position because we sell directly to enterprises and there is an amount that they're willing to to pay for a service like this. But then it still comes with some restrictions. And typically what we found is when you scale by query volume. So if you're using some API, if you're only using an API and you scale by query volume, it's fairly tractable cost wise. But then when you look at okay, do I want to do some processing with an API on every single document, some of that could be tractable too, but it becomes a little bit harder to reason about, especially when you, when you go to really big enterprises with big hundreds of millions or billions of documents. But on the other hand, there's the whole aspect of rolling your own LLM and we have amazing talent at glean to be able to do that. And when you roll your own LLM there are cost structures that become a whole lot better and it is far more tractable and effective and many people.

Are working on it.

But the problems around permission and security that Eddie highlighted are really the more challenging bottlenecks. I don't think cost will be a bottle and it does come down with time as with any sort of new innovation.

Sure, sure, yeah, I'll just tack on to that. I firmly believe these costs are going to be driven down. There's A lot of really good, healthy competition happening. We've seen it in recent days with some of the larger companies, but even before that you have a really good group of LLM providers all kind of pushing each other to be better and ultimately the consumers win. When this kind of stuff happens, the costs get driven down, you get this as a commodity, and more value is created.

So often on the show we like to land on the sort of forward looking question, where do you see this technology going in the next five years? How do you see it impacting, you know, the specific work you're doing, your industry at large, that kind of thing? Talking about generative AI and LLMs, it almost doesn't feel fair to ask you to think that far in the future, five full years out, because everything is, is moving and changing so quickly. But how do you see glean in particular making life easier and better for your customers in the near term over the next couple of years? And if there are things coming down the pike that you can speak to and you want to share, great. And if you want to speak more in abstracts, that's just fine too.

I can take this first and then maybe Eddie can pitch his view right after. I don't have much to add on about the glean use case. I think it's fairly clear it's that Slack bot or that that vision of not having to rely or waste time finding information and LLMs go a long way towards addressing that need. But to me personally, honestly, I'm more curious about how this unfolds for the rest of the world. And it's really crazy. You know, bing announced this ChatGPT integration yesterday. Google announced Bard. Even consumer search as we know it since 2000 is not going to look the same five years from now. And I wonder what that means for the next generation of people growing up who just aren't even tuned to having to parse through and decipher what's real from what's not. They just get an answer back and they trust it or not. I don't know what that looks like and what that means or what the implications of those are, but I do see the landscape being very different For Search Overall 5 years from now than it is today.

I'm not trying to hang you up on, you know, to catch you, so to speak, on something, Didi, but. But something you said caught my ear. You said we're talking about this generation of people growing up with these tools and asking an LLM a question and getting an answer back and trusting it or not, why Wouldn't they trust it? Or do you see a situation where you have a filter that can kind of give you a little alert icon if it's maybe not trustworthy? Or do you think it's just an inherent thing that when a generation grows up with something, they kind of intrinsically learn, you know, when and how to trust it and when not to.

I think it is a generational thing. You know, you can do the same thing with search. We know that if you ask a question that proposes a fact that's clearly not true, then the likelihood of today, Google or Bing telling you, yeah, this is an article which verifies what your completely absurd statement is true. And LLM somewhat work in the same way. But, you know, I trust humans. We figured out what's true and not true with search. And, and for LLMs, even today, people can accept pretty easily that this might be 90% true, but sometimes it's not. And that's okay. It just tunes your internal meter of how much to trust something you read. And I don't think it means that there's going to be misinformation spread everywhere necessarily. But humans do adapt. So I'm not so concerned about how that looks like for the next generation.

Right. The invention of the printing press, you know, theoretically helped truth just as much as it helped lies. Right. The press doesn't care what you print on it. It's a, it's a tool.

That's true.

Eddie, thoughts on the LLM driven future of search and productivity?

Yeah, I think, I think Davey's points make a lot of sense. I think it's kind of an open question, even not even just in a few years, in the coming months, how users will respond to the products that are built, how those products will be built, and how, you know, we will look at the output and build that. Oh, I'm 90% sure this is right. And so I can work with that. How do product builders kind of responsibly set that expectation while researchers are kind of driving the improvements that actually take that 90 to 95 or 99 and what have you.

Right.

And so it all comes down to delivering this technology in the right product. Right. Without which, you know, it's, it's, it's not going to create value. I do think just to kind of emphasize one of the things Didi said about how fast people growing up, the way they interface with technology, how fast that changes and what expectations they kind of have as they go from being in school to being in the workplace. There's this really Interesting article. I saw a few months ago about how people, kids who've grown up kind of in the G suite set of tools, right, using G Drive and stuff, they don't even understand why that there are folders. They have no concept of a directory structure or a files being in certain places. They just search to get to everything. They just, there's no organization needed. They just know that they can search to find what they need. And I mean, obviously that's, that's great for us at Glean, for all those people that you're in the workforce, but I do think that just a testament to that, that kind of shift in how people interface with technology. And as product builders, we have to be responsible about working with that set of users and making sure they understand what they're using and they can continue to get value out of it.

You know, without giving away my actual age, I will say it made me think of how. I think my parents think of me as the power computer user and my kids probably think I have no idea what I'm doing on the computer because I'm kind of doing it, you know, the old way, but, well, but then they're doing it this new way because it's, it's a native tongue to them. So that's, that's a, that's a great point. Fascinating to think about. Well, guys, Didi and Eddie, this has been, this has been tremendous. I feel like we covered a lot of ground in a quick amount of time, which makes me want to make a bad joke about LLM powered search covering a lot of ground, but I'll, I'll skip that and get to what people want to know. I'm sure people listening are going to want to know more about Glean, the company and your products, but also some of the research that you've been doing you're sort of referring to and just everything going on. Like we said, it's such a fast moving space right now for folks out there who want to learn more. Where would you direct them? Online.

So for Glean specifically, you can go to glean.com glean.com blog and/careers or if you're looking for a job and, and if you want to reach out to me Personally, I'm@ddlean.com or you can find me on my Twitter or LinkedIn.

Eddie. Yeah. Glean.com that's. I live in. I live and die Glean. So I got to point you to.

I love it. I love the dedication. Again, guys, thank you for taking the time to come on this is great. I look forward to the chance whenever it comes for me when I get to Triglyne, for myself at work. And in the meantime, all the best with everything you're working on, the research, the products, the company. And let's stay in touch. I'm excited to hear what you're all going to be up to, you know, the next year or two down the line.

Absolutely.

Noah, thank you so much. Thanks so much, Noah. Great to meet you. Great job today.

Sa Sam.