Is search dead? Google vs ChatGPT, from Google Search to enterprise search at Glean, machine learning in search, tech layoffs - Deedy Das - The Data Scientist Show #061

For ranking, you need to have a lot of trust in the engineer to be able to, on their own, dive very deeply and understand data and what's going on.

Yeah.

Because if their mindset is I just want to ship this project, you're going to have a terrible time. It's your job as a search engineer to say, hey, this can't ship. Even though you wanted to tell your manager, even though you want it to ship, everyone wants it to ship. It can't ship because of this reason and this metric is not working. We have to dig into why this isn't working. Even though we had this hypothesis, it didn't really show in the metric. We wanted another metric went up and maybe you could launch with that metric, but we don't understand why it went up. So we should dig in more. All of these are kinds of. It's a thought process that search engineers have to have that's very different from.

Hello everyone, welcome to the Data Scientist show. Today we have Didi. He's a founding engineer at Glean, an enterprise search startup. Previously he was a tech lead at Google Search, working on query understanding and sports products in New York, Tel Aviv and Bangalore. Previously he was an engineer at Facebook and he graduated from Cornell University. He spent two and a half years in his undergrad and finished his master in computer Science in one year. Specialized in machine learning outside of work, Didi writes on his blog. He also enjoys running marathons, road cycling and playing cricket. Today we'll talk about search projects he worked on at Google, why he left Google, his current work at Glean and his thoughts on whether Google is doomed because of ChatGPT. If you enjoy the show, subscribe to the channel. Give me a five star review. Welcome to the show, Didi.

It's great to be here. Daliana, it's pleasure. Thank you for that lovely intro.

Yeah. So you worked on search at Facebook and Google and now joined a startup working enterprise search. Why search is so interesting to you?

Yeah, it's a great question to start. So my beginning with search sort of all has started from the beginning of my career. I started at Facebook when I started out in New York and I spent a year there and that's when I first got a taste of working on search in the industry. But it never really fulfilled me. I think the first time I remember using the Internet, the magic of computer science was always Google search. The fact that everybody could find any information in the world and that's still 25 years later, not a solved problem. And it combines so many of the unique aspects, the fundamental computer science problems in one, natural language processing, information retrieval, distributed systems. And it just seemed like the perfect problem to work on. And that's how I got into search. And when I interviewed at Google, I was very clear that I didn't need to leave Facebook and I only wanted to join if it was search. And they finally agreed. So I worked on search at Google for about four years.

Yeah. And let's talk more about your project at Google when it comes to career understanding. So when you started working at Google, what is the difference between the real world search problems versus what you learn in school?

Well, search in itself, the first thing you have to say do is clarify what search is. When you're on a webpage and you're pressing control f and you're searching through the page, and that's one form of search. When you're searching through a small data set on your computer, that's search too. But no one really considered those very hard problems or very fascinating. The real fascinating part of search really is when you get to web scale, when you get to billions, if not trillions of web pages and you're able to, if we step back and think about it, you're able to scan through trillions of web pages in under a second and tell the user, hey, these are the most relevant things to what you searched for. So that's what really separates search in the industry at Google scale versus search at any other scale. And mind you, Google is also one of the oldest software products, modern software products that exist. So you're also working on a code base which is 25 years old. And that has its quirks too.

Yeah. So when you worked on query understanding, what was the challenge? What was the problem you're trying to solve?

So the high level statement of query understanding is, okay, in search, a user comes to Google and they type in a query. At that point of time it was also assistant. So people could also say queries, which kind of changed how, what, how queries looks like. But for the most part you're operating with a string, which is what the user wants to search for. Now, computers don't understand strings. They have no meaning.

Yeah.

So to be able to serve a query well, you have to understand what that string means. And this could be something as simple as, okay, the user search for Obama. Is that Barack Obama or Michelle Obama? It could be as simple as that, or it could be a little bit more complicated. What do you mean when you say, is Barack Obama president? How do you structure that in a way that a machine can understand. So query understanding really involved, okay, how do we get from string to a parsed representation that computers can understand? And then two, given a bunch of parsed representations, how do you rank them? And then three, given your top ranked parsed expression, how do you meet that need? And that has many, many, many problems that go with it. But fundamentally that's what the query understanding problem is.

And how do you solve this problem?

Well, it's really hard to say in a couple of minutes, but by and large you learn, you come up with an ontology of representations. And so many people may have heard of the knowledge graph. Yeah, the knowledge graph is, you can think of it as a graphical database of every single entity that exists. That's step one, you need to have a representation of knowledge to be able to even get Obama right. Step two would be to understand, okay, now how do we get from a grammar to a question? And that involves defining an ontology of okay, what kind of questions are there? How do you structure them? So it's a very complex set of procedures. But fundamentally you do that by understanding the base of knowledge, developing a schema. What are kinds of questions you can ask and how can you compose them? You can even have more complex questions. What is the height of Obama's wife? Yeah, that's a more complex question. And you're doing two jumps in the knowledge graph and further and further iteration on that. So you can go on. There's analogies in sports, general people, movies, when is a movie coming out, and so on and so forth. And everything you see on Google that comes on top, that's not a web result, is typically something that's answered by Google's knowledge graph. So a very precise query, understanding and then understanding what can answer that query.

Okay, yeah, that makes sense. And like you mentioned, people might ask them specifically specific questions about president or some current events. Do you need to hard code some information to make sure those are accurate, especially when it's prone to misunderstanding?

Well, on some occasions, yes. It really depends. Google tried to build a system that's very resilient to this, so you'd be surprised how infrequent that is. It does happen. But suppose there's not a good example. I have top of mind, but suppose you have a name of a person that now conflicts with a name of a movie. So okay, here's a good example. Suppose you query for Harry Potter. Harry Potter can mean the whole novel series. It can mean any one of the novel series. It can also mean the movie series or Any one of the movie series. And if a new Harry Potter movie comes out, you don't want to hard code the new movie in your algorithm should be able to understand, okay, this is fresh, this is something that people are looking for. And this is probably the most likely representation that the user means when they say Harry Potter. So if a new movie comes out, it's probably going to answer with that movie instead of an old book.

Right. And because the search results will impact a lot of people's, a lot of users experience. So when you say update a model, how do you test the results? Do you have a team of people to do some manual check to make sure it covers some edge cases or like new movie updates?

So Google is pretty public or not pretty public. Not many people know about it, but they are public about how they do their result testing. By and large, the approaches, I or an engineer makes a change to ranking and then they run what they call an eval or side by side. So they say, okay, this is what ranking looked like for a given sampling of queries before my change and this is what it looks like after my change. Now they send those side by sides to a group of raters and Google hires a bunch of raters, contractors in many parts of the world who have to read a very long handbook of how to rate search results.

Oh, okay.

And then they rate search results and they tell you, okay, this is relevant, not relevant, highly relevant. It's a very complicated scheme. And then you parse those together, do some processing and figure out whether your change is generally positive or not. However, I will preface that by saying it really, really depends on the launch. This is by and large coverall kind of approach. But there's many other approaches Google uses. Not every approach can be, not every launch can be covered by this. So they do live experiments, they do AB testing, they do different kinds of engagement metrics. Sometimes it's not always just about surfacing more results to the top. They look at how many of these results are duplicate results, page diversity. They look at what are the quality of these results. There's many things that go into that. But high, high level, yes. It's about how raters rate the quality.

Of the search result page and the quality of the search results for different type of model. I guess also for Google, different stages of the business, they can prioritize different type of metrics. I remember when I worked at Amazon for different team, they prioritize different things. Sometimes they want to make sure people click on the top three, five results, which is A common metrics people look at for some other products, maybe they want to, they prioritize high quality products. They want people to click more expensive products or products with higher review. So what are some common metrics you have used for your models?

So at Google there was a metric, I'm forgetting what it was called. There was one top line metric from for information satisfaction, I think that was what it was called. Yeah, that was one. And all the metrics you named, there's various NDCG metrics, there's various metrics on, like I said, diversity. So how many of these results are actually different from each other? A lot of relevant results aren't helpful. Then when you run a live experiment, of course you have metrics like ctr, hover time and so on and so forth. So there's so many to name. And every launch, as you rightly said, is decided by its own set of metrics. It's not one size fits all.

Gotcha.

As you can imagine, an image result has very different metrics than clickable result and you can't really compare those apples to apples. You have to figure out what does it mean for an image result to be good versus a result which has to be clicked on.

Right. So you have to be in tune with a business use case and users to define a success of your model.

That's correct, that's correct.

And it's an interesting timing with this recording. Yesterday Google just launched bard, the conversational AI powered by Lambda so they can compete with ChatGPT. And previously you wrote a thread comparing Google and ChatGPT with this new launch. Google is not as quote unquote doomed as people previous think they are. So what do you think about this launch and how do you think this will change the current landscape on conversational AI and its impact on search?

Yeah, it's a fantastic top of mind question for everybody. I think the first thing to be said is for anyone who thought Google was doomed, they clearly don't follow the research very much because the fundamental model that powers all of ChatGPT is based on a paper called Attention is all you need, which is the first paper to introduce this architecture of deep neural nets called the transformer. And that paper was written by a bunch of Google researchers. A lot of the contribution to the large language model space is by Google and they are well on top of GPT in terms of any of the traditional academic benchmarks with natural language processing. They have a model called Palm, they have lambda, they have flan T5, they have a bunch of different Models with different approaches. They have DeepMind, who has their own models. So I'll just preface it by saying Google was always on top of this. The real question was how quickly can Google act to go from research to actually productionizing this? It's a bigger deal for Google. Search is their entire business. It's 70% of their business. To do such a launch hastily on search is probably not wise. And people say ChatGPT has 100 million users today, but that's very, very low compared to Google.

Right.

So that's my preface. Now, how do I think that'll affect the search landscape? Well, it's very clear that for a lot of queries, I find them valuable. The whole world finds it valuable. ChatGPT does really, really well on a very large set of queries. It's able to. Instead of the paradigm of where a user searches for something and has to click on a bunch of results and figure out what the results are saying and what they want to trust and not trust. ChatGPT might be wrong sometimes, but it'll just give you one answer that you can read and make sense of with coherent text, unlike most of the Internet. So it does very well, but it doesn't do well on everything.

Right.

So to just highlight a couple of problems with it. One is it's not fresh.

Right.

So everyone has run into this. Probably it says I can't answer questions past my training time of whenever. December 2021. Yeah, so that's one. Two is it hallucinates a lot. So sometimes it's wrong. Searches can never really be wrong because it's just directing you. 3 is it costs a lot more than search today. It's going to come down over time, but right now it's pretty expensive to train and then launch and even the inference cost is fairly expensive to run these things. And then four, it is a lot slower than search two. And the final point, and that's the point I want to drive more deeply, is for any query where it's where Google would at least call it a head query. So a query like Trump, you want to know the latest news about Donald Trump or and facts and figures about him. ChatGPT is probably not going to do a very good job at that. Or you want to know a sports score, it's a very specific information. You want to know the score, who scored how many goals and what's the timeline of the match. You want to see a nice rich UI to explore that. You don't want a result, you don't want to see 41 to 27, even if it is accurate just in text. So I think for head queries, ChatGPT is far behind Google. But the reason people are so fascinated is because in the last couple of years, Google has been really bad at tail queries. People say that you have to append Reddit to the end of a query to find what you're looking for. You could never find anything authentic if you were looking for, I don't know, hair dryer reviews for a particular brand. You're never going to be able to find anything good on Google. It takes a while, but ChatGPT will at least tell you, hey, this is good and bad. This is why it's good and bad. It'll just summarize it for you. So I think for that set of tail queries, which is a big set, ChatGPT does a lot better than Google. And maybe with Bard, Google will be able to start tackling that use case.

Now, with this kind of competition between Google and Microsoft, do you regret leaving? Google wouldn't be able to kind of participate in this exciting race.

I think yeah. A lot of people in the valley have this view that big tech companies are boring and move too slow and not great. And I understand a lot of these criticisms, but a lot of that stems from the fact that startups are fast and more exciting and they are, but they underestimate often just the amount of impact you can have at Google if you really do care about the product. There's a lot of cool things that are happening at Google right now. And so, yeah, there's a part of me that wishes I could do a lot of things. So, yes, I do wish sometimes that I could also do that.

And what are some interesting experiences or unexpected story that happened to you when you were at Google?

An unexpected story at Google? Well, there's one that I talk about a lot, which is my second project, which we haven't talked about yet. But on the latter half of my time at Google, I shifted gears from query understanding to working on a full stack product. And that full stack product was something very close to my heart. It was sports at Google, which is in itself the biggest sports app in the world. One billion users, yada yada yada. And particularly I was helping build out the cricket product, which is something I'm really passionate about. And when we were doing that, it turns out Sundar Pichai was the CEO, was a big cricket fan, and in particular this one instance that was quite funny was we had just moved, we'd done a very Long evaluation of how to move to a different data provider that would make sure scores were much faster and it would appear on Google almost instantaneously. But it still had some. It wasn't completely refined. The provider was new, they weren't an established player. And when we just moved to them and we launched and we had our first big game, the provider just conked out and gave us all these bad scores. And on live on the Google page, we were just showing the worst scores, that it wasn't just inaccurate or slow, it just didn't make sense. It's not a valid cricket score. And Sundar filed a bug straight saying, what's going on here and why is it so bad? Can we turn this off? And so we fixed it eventually and we, and we recovered. Sports is a very interesting team to be in at Google because unlike many of the other teams, you are deadline driven and you are on call whenever there's a sports game because if things go wrong, a lot of people complain.

Yeah, that's such an interesting story. Sometimes you don't know which VP or CEO in a company is dogfooding your product. There's a similar story in the previous team I worked on when I was at aws. Andy Jassy at that time he was the CEO of aws. He haven't took over the entire Amazon yet. He's a big hockey fan and, and our team was working on one of the hockey leagues I think to do some video analysis. And he was very passionate about a specific project. And then it's very. It motivates the team a lot. Right. When you have someone from the VP level looking to your project. But like you mentioned, it can be a little bit stressful at the same time, it's high visibility. In the end, it makes a good story.

I can tell you that I was motivated enough already. I was just stressed.

Yeah. So when you moved to the sports product team, was it your decision?

That's actually another very fascinating story. I think what happened? This is what happened. My manager at the time in New York knew that I was into sports and I brought it up a couple of times. I'm like casually said, hey, this doesn't work so well and this doesn't make sense on search. And he said, he's Israeli. So he said, oh, that's really funny that you say that. I know the guys in Israel who run this product. Yeah, at the time it was a team in Israel. And he said, hey, Google has this concept of a 20% project. So he said, hey, why don't you? If you're interested, I could send you to Israel for two months, to Tel Aviv to work on cricket for 20% of your time for two months. So I said, yeah, why not? It sounds like a free vacation and a thing that I'm really passionate about. So I did that. And in those two months, I loved working in Israel for many, many reasons. Like, that's a whole podcast in itself. And the team there said, hey, do you just want to stay and work on this? Because we really like working with you. You did some good work and we'd love to continue this. And so I did. For a while, obviously it wasn't sustainable, but just legally for somebody who's an Indian citizen to be in Tel Aviv, even though I have a visa in America, didn't work out so well. And then strategically, the stars aligned and they said, hey, would you rather go to India and start another fork of the sports team? And so I eventually did that.

Yeah. How about you just give a few quick bullet points on why you love working in Tel Aviv? What is the culture compared to New York?

So. So the one I always start with for tech is unlike any other office I've ever worked in before. It is the only office with close to a 50% gender ratio. So in itself, the energy is much more lively. Usually when people think about tech companies, people are pretty quiet and introverted and kind of busy in their own work. And the cafes have a little bit of chatter near the coffee stand, but that's about it. But the Israel office is lively, so it's people calling out to each other across the floor going like, hey, help me with this, or hey, do you know why I'm seeing this error? And it's just, it's a good energy. People do stuff outside of work together. People are much more friendly and casual. It's not a very formal workplace. So that's one thing I liked. And the second big thing is culturally, Google makes you do some cultural training when you go to another country. And one of the things they said was, be careful if you're American and going to Israel. People tend to be straightforward, so don't be offended by whatever they say. And in my head I thought, look, I'm Indian and straightforward does not offend me. It offends Americans, perhaps. So I loved the fact that in Israel they are not just straightforward, but there's a clear separation between, hey, I'm giving you my honest feedback on your work, but don't take it personally. So it's not said out loud, but it's very clear in how they behave. And I really like that because one of the things I don't like about working in America is everyone is dancing on, dancing around the main issue all the time. They're trying to be politically correct, they're trying to be nice. And sometimes you just need someone to say, hey, your design, it's awful. It doesn't make any sense. We should never do this. And this is why. Yeah, because you can react to that. In Israel, they would do that, but they wouldn't. It would never seem like they're trying to make fun of you or pin you down. It was their honest opinion. So I liked those two things about.

Yeah, I'm from China originally. When I first moved to the us Also have the challenge. I slowly learned when I disagree with someone, I have to tone it down. I cannot just say, hey, you're wrong. You have to kind of sugarcoat a little bit, like sandwich it or whatever. And also challenging for me, when my manager were giving me feedback, they will kind of hide the feedback in some compliments. And it didn't resonate with me. I didn't realize there was a problem. And later on they have to be more specific about it and they're like, oh, okay, you're saying that part wasn't good enough. And I have to kind of really listen to catch that. And I would rather someone just tell me, hey, that's bad, that's okay.

It's so hard. I was blessed that my manager was Israeli too, so he would just very honestly tell me stuff. But I've been through that situation as well.

Yeah, so. And later you left Google. You wrote a blog post on why you left Google. One thing you mentioned I thought was very interesting, your coworkers would say, if I wanted to work hard, I'd go to Uber or Amazon. Do people really say that?

Yeah, I mean, look, Google is notorious for, amongst all big tech companies, being the most chill. Maybe not the most politically correct thing to say. But there's plenty of people at Google who hardly work 40 hours a week. And until recently they've been layoffs. And the culture is, I hear it's changing and people think differently. But for the longest time you could sit there, own one service for 10 years, update it whenever you wanted to, and no one would ever get rid of you. It's really hard to get fired at Google for non performance. You can get fired for other reasons, but Google just never ever fired for the longest time. So yeah, it is true that the culture at Google is not to work very hard. I can say for sure from Facebook it's a little bit different. They definitely have more of a focus on shipping more code but I think it's more quantity and not quality. But I had friends at Amazon and Uber who were spending longer hours at work to do what they had to get done.

Yeah. Also had friends worked at Amazon later on worked at Google and they told me it's true.

Yeah.

So what are some other reasons that made you decided to leave Google?

I would have to struggle to remember what I'd written in the blog.

Oh, I can't remind you. Other things you mentioned speed of execution. I think it's what we talked about. And also high level strategy. Google tend to play safe internal tools. I think this one I can relate to. Also in Amazon or any other big companies you have so many internal tools, you just learn internal tool. You don't know what are some other new say data or machine learning tools out there. So there's like a disconnect and whenever you want to bring some open source tool you have to convince the compliance some legal and it's just really hard to kind of innovate and try different type of things. So those are the things from your blog. Do you have any other things to add?

No. Broadly, I think to hone in on two main points. One is my frustrations with Google at the time and two was things I generally had always wanted to do. I think to start with my frustrations that Google is for good reason a pretty slow moving beast. It's a large organization and it has to be very careful or things can go very wrong. And you can break a lot of things, especially in search. But it come to a point where it felt like I was always fighting management to try to get the right product out. And I strongly believe in building the right product. And it's not like I think my way is my way or the highway. I wanted input. I wanted to just move fast so we could all agree and get to a better place. But at Google it's really hard to move fast when everyone else doesn't want to. And so there was that frustration. The levels of approval. Everything has so much red tape to get even a small thing launched. It has to go through like three directors and a vp which is kind of ridiculous for some of these things. So that was one Google is just slow and it wasn't moving fast and too much red tape. And then the extrinsic reasons are things I generally wanted to do. I felt like I was losing touch with the industry like everybody is talking about these new hot tools that do these new things and Google is its own little echo chamber of oh but we're Google, we don't need these open source stuff, we're better. And the more you look into it you realize you're actually not better. You might have been 10 years ago, but not anymore. And so I really felt like I was losing touch with what the tech, where the tech industry was going and if I kept doing that I would be stuck at Google for the rest of my life. So I thought it was a good time to explore something radically different. So I joined a very early stage company.

I would love to learn more about what you do at Glean. But before that, for people who are interested in getting into the search space or want to say more specifically, join a search team at Google. What's your advice to them, what kind of domains of knowledge they need to learn, books you recommend?

Unfortunately to get into Google you don't need any of that. So you just need a good interview and hopefully you get placed on the right team. Unfortunately the process is I think pretty broken. But if you are curious about search and how search works, there's some great I don't know about a book because a lot of this information is really hard to find.

Yeah.

To go on a little bit of a side bar there is, there was a time when academic literature never evolved for search because Google was the only company that capable of actually running experiments at real world scale. So you can read some of the early papers that people at Google wrote. So if you Google like just research papers on Google Scholar and go back to 2005 to 2010, there's some exceptional papers on how to build large scalable multi level hierarchical inverted indexes and execute Boolean queries. On top of them there's many, not so many on ranking because that's a lot of secret sauce. Yeah. So that would be a really reasonable place to start. And a lot of ex Googlers have written pretty hard to find funnily enough because it's really hard to even Google for. But really good blog posts and engineering posts on how they built a lot of these systems. There's one, Russ Cox is a good guy to look up. He has some exceptional posts about it. Yeah. So that's where I would lean on now. A lot of the public information is elastic as a repository. Just if you understand and learn elastic and all its little quirks, you will learn a lot about the fundamentals that go into search engine. Probably not at Google Skill but you will learn, I think 80% of it.

Yeah. So information retrieval and elastic search.

Yeah, Information retrieval papers from Google authors back in 2005 and Elasticsearch would be pretty good sources.

Yeah. So you're saying the information retrieval if back in 2006 things hasn't changed that much since.

Well, that's not what I'm. Well, the fundamentals haven't changed that much. What really changes from then is really okay, now let's add knowledge and query understanding. That's something that's a whole separate topic outside of information retrieval. Search is a thousand topics in one. This is just core information retrieval is what I'm talking about. And ranking has changed a lot from 2005 to 2020. It's really hard to find papers on that though because it's a. It's all private to Google.

Yeah.

So I think a lot of things have changed. Ranking knowledge, query understanding, yada yada yada. But it's just really hard to find the source of that.

Right. Gotcha.

Algolia has some good blogs, Elastic documentation. Engineering blogs are very nicely written. There's a bunch of really good resources that explain the basics.

Yeah. And also I think it's a good source to learn is to go to tech companies engineering blog. Sometimes they'll publish papers on how they solve some ranking problems.

Yeah, it's true. I would say that in my experience, most of the tech engineering blogs talk about scale more than ranking because unless you're a search company, it's almost pointless to invest too much effort in ranking.

Yeah.

So most of the ones I've read have been like how we scaled to index x thousand docs in two seconds and to serve 20 million users instead of how we improved on ranking. Although recently I read one from GitHub Code Search. Fantastic blog post about how GitHub Code Search is designed and works.

So now you joined Glean. What is enterprise search versus consumer search?

Yeah, that's a great question. So the way I like to talk about it is to outsider or layperson, it might sound like you just took a problem and made it a lot easier and you're solving the easier version. But I think enterprise search is a fundamentally different problem. And there's a couple of reasons why that is. One, well, some evidence of this is that despite Google being so successful with their consumer search, they never managed to make a dent in enterprise search. Even though they have an enterprise search product, they have Google cloud search. It's not as big as you'd think. There's a couple of reasons for that one is in enterprise search, you're solving a very different problem. Permissions don't exist in web search. Permissions definitely exist in enterprise search. Freshness needs in web search and enterprise search are I would say fairly similar. So that's not really that different. You have to have it in order of seconds. Let's go to another one. Security in enterprise search is very different. That's not a very interesting one to explore. It's just yes, it has more security implications. The one really interesting one is a lot of consumer search ranking is tuned off of feedback data. It works because there's billions and billions of people who are searching for things every day and clicking on things and you're analyzing that and making the ranking better. Now in enterprise world, two things are different. One, there's not a lot of the distribution of queries is much more long tail and flat because most people are not searching for the same thing in a company like maybe you can come up with a query like benefits, which maybe most people may search but in practice usually not the case. But most people are concerned with their own projects and they're searching for information that they need. There's not many like Obama type queries in enterprise. So that's one. And number two is the volume in an enterprise is much, much lower than in consumer because for a given company, even if you take a 100,000 person company, if every user in that company issues 10 queries a day, which is pretty, pretty high, that's a million queries a day. On completely distinct queries you might see a million non unique queries and still see maybe 500,000 unique queries. So each unique string has maybe one or two data points, if that. So it's really hard to train a good model off that kind of data. That's another point. And then kind of related to the. I can go on and on but kind of related to the security point, you have much less visibility into what is right in an enterprise. For example, if I were to, if I was a company, if I was glean and I deployed on your company that you're working in now and a. Do you even. How do I debug. Do I see your results right? Do I get to see them or not?

Yeah, that was my question. Like how you have to do some fine tuning when you deploy because the, the contact like you mentioned is very different. But then there's also this permission security issue, right?

And it's a really hard problem. And even in the cases where suppose you as a customer say in Some cases I'm happy if you look at and debug my queries. Suppose I can see some anonymized stripped down version of it. It's very hard for even an engineer to say if this is good or bad. I have no context on what you're working on. I'm looking at a bunch of results and I think that looks reasonable. It has the couple of words you were looking for and it's ranked pretty high. How does one even begin to approach the problem of understanding how to deconstruct that and make it better? So these are all very hard questions and I can't say too much otherwise it'd be revealing our secret sauce. But that's why enterprise search is a very distinct problem from consumer search. So the things that work in one don't necessarily necessarily work in the other.

Not sure if this is your secret solve but for example, the problem we just talked about, if there's some specific issue, if you need to help them debug. So how do you solve this problem for different customers? Different type of context.

It really depends on a customer by customer agreement basis. Again unfortunately the more nitty gritty part of enterprises it is a business deal at the end of the day. So you write a contract with the customer.

So it's different.

Yeah, yeah. That allows you. That tells you what you can and can't do.

And you have probably like solution engineer to work with a customer.

Yes, we have solutions engineers.

And at end of the day do you want us to be a completely self service like SaaS product or it's always going to be some type of support from your side so.

So that's a great question. I do want to lead like this might go into a slightly different topic but a lot of people say they do enterprise search. In fact, enterprise search is not new. People have been trying to do enterprise search even before Google. People have been trying to do it since the 1990s. People have always had information needs at the workplace. What makes us believe one of the reasons we think we're doing it better is because a lot of versions of enterprise search just don't tackle the right problems. Either you're searching through your own documents or you're searching through one subset of the of the company's public documents or something or the other. Whenever we approach customers today to talk about glean, a lot of customers just don't believe enterprise search can be this good. Yeah, their first reaction is now and we've seen enterprise search for 10 years and it's all been Terrible. And then when they try our product, they're shocked. They're like, how is this? This is exactly what I wanted. But none of the others suited your needs. So what makes us different? Yeah, we can't actually go. To answer your question, we can't go self serve completely. Usually there's some ifs and buts because when you're deploying at a company, we need to index the entire company's corpus. To index the entire company's corpus, typically you need a sign off from various owners in the IT department for G Drive. Who owns G Drive in your company? Who is responsible for managing keys and accounts that can link to G Drive? API access to G Drive. What kind of access do these products that link to G Drive have? That usually goes through pretty rigorous security review. So we rely on sort of a top down deployment to make sure our product works well. And so can it be self serve? Perhaps for small companies where this doesn't matter so much that they just want to say plug and play and connect. But typically for large enterprises, I don't think even though our product can support it in practice it will require as many enterprise products do, multiple meetings and set up and conversation and a back and forth before we can deploy.

Gotcha. That makes sense. Enterprise search and consumer search is different. And in terms of the technical skill set, how do compare the engineer skill set at Glean versus what you need to know when you are at Google?

Well, a lot of Glean engineers came from Google so it's really hard to make a comparison. I do think that, look, at the end of the day, smart engineers who have some domain knowledge can typically adapt it to a slightly adjacent thing. So which is why we are able to do this. I don't think it would be a little, it would be hubris to say that we're just so much better than all the other Google engineers. There's a lot of Google engineers who could probably understand and do this.

Yeah.

So I don't think it's that different. The problem is different, but the techniques, the ways to think about it, the ways to reason about evaluation techniques and iteration and what kind of signals work for ranking and don't work and what do you. All of those things are common between anyone who works on a search product.

Yeah. And if you feel the technical skills are kind of transferable, what is the moat of the business? What make you special?

Well, I don't think any business's moat is technical skills. There's none. I mean everything that OpenAI can do today, most other People can do. Yeah they know the techniques. The hard, the the moat of our business really is how good the product is. Someone will still have to rebuild all of this stuff that we've thought out and built and we've deployed to a bunch of customers. We know how good we are on those customers. Some of our customer reviews are through the roof. So in order to compete you're welcome to everyone's welcome to start. So it's co capitalism works. You can start a company, you can try to reverse engineer the secrets and try to make it work and we'll see who innovates faster. So I think the big the difference and maybe this is the way you wanted to or what you're curious about is there is a mental shift because when you do move from Google to a startup you have to think faster. You have to think on your feet. Our founder happens to have a strong background in enterprises and so he knows how to do enterprise sales. He knows how the motions of this business work. So a lot of things have to come in place for a product to work. It's not just hey I developed the right algorithm and therefore mine should be successful. So yeah, it's a long winded answer to a pretty simple question.

And have you thought about adding ChatGPT or other language models to summarize some results?

Yeah, we already made one and we are rolling it out right now. So there that is a part of our plans. I don't think that there's an easy way to do this and there's a much more a better way to do this right now as all the problems I listed still exist. It's slow, it relies on these results and it summarizes them but those results, that ranking itself is the glean secret sauce. So we think we can do a really good job because a lot of the result summarization relies on good ranking and we think we have really good permissions aware ranking and so yeah we are doing it and I don't think that it will be easy for other competitors to do it easily because of some of the difficulties with deploying LLMs in the enterprise and I'm happy to talk about that separately. And that's not the only thing we're doing. We're working on many other ways to use large language models to make enterprise productivity so much better.

Yeah and so when you think about deploying your product into different companies and different company have different data business use case when do you decide whether you need to update a model for a specific company or you do this In a more centralized way, we roll out.

Two times a week to all our.

Customers all the time. Yeah.

Yeah.

Is this gonna be, is this going to be the cadence kind of for a while?

Yeah, I think so. For the foreseeable future. There are issues with it. I mean, there is, it's really tough to have biweekly rollout cadence when you have many, many customers. Especially when you're managing something as intricate as search. Things can go wrong really quickly. But yeah, There are some SaaS companies that don't do it that way. I think for search we are required to because if we want to be able to improve and learn from all of these, all of this customer, how customers use our tool and get better, we need to keep that cadence going. Iteration is key to building a better any ML solution, but specifically a search solution. For sure.

Yeah.

The less frequently you iterate, the less data you get and less you understand and the fewer things you can really do to make it better. So our search is always getting better. Yeah.

And for people who got laid off, what kind of advice do you have for them?

A couple of tips that I think are quite helpful that are a little different is a lot of people who've left big tech companies just don't know how to evaluate startups.

Yeah.

And for them, I highly, highly recommend talk to a friend who works at a startup, if not anything else, that's first one, just to ask them how should I be thinking about this? Because that'll already give you a lot of context. Two is, if possible, I would try to get in touch with a VC. People don't really understand this, but VCs have an interest in placing engineers at their portfolio companies because it's the right thing for their investment. And so if you reach out and you're talented and the VC wants to spend time with you and you're honest and you ask, hey, which one of your portfolio companies do you think is the best? Oftentimes they'll just tell you and they have insight into those companies that very few other people do. They go to the board meetings, they understand this stuff. And so it's a very good way to find the difference. If you just go look at a list of startups, you're not going to be able to sit, tell what's good from bad, what's right from wrong. So this is one of the good ways to do it. And I would last thing I would say is don't rely on hype to join a startup. Like that's just a terrible way to make A decision. If you use hype to join a startup, you might join a good company but for all the wrong reasons. So do an analysis, understand a little bit of business and find out from these two techniques what kind of startups are worth joining.

Yeah, that's a good advice. I never thought about ABC could be a good resource going back to your startup. When you work on a machine learning project, do you mostly hire software engineers, have backgrounds in machine learning in ranking the specific problem related to search, or do hire data scientists or kind of generalists to work on those?

We have an approach that's very similar to Google, so we would hire people who have a mix of. Well, let's start with everyone who works on our ranking and intelligence and search teams must be able to write and deploy production code. So it's going to be somebody who is an engineer, but they can come from various different backgrounds. So they can come from a researchy background but have some engineering skills. It can be a data scientist but has some engineering skills. It can be an engineer who's done applied ML in a previous role. We have our own interview process to make sure we try to tease these things out. Personally, I do think the skills involved to be a good ranking or intelligence engineer is a little bit different from normal engineering. I think a lot of normal engineers are in the cadence of what do I have to do? I'm going to build it, I'm going to measure it and if you're infra measure some latency and then say it's done. For ranking you need to have a lot of trust in the engineer to be able to on their own, dive very deeply and understand data and what's going on. Because if their mindset is I just want to ship this project, you're going to have a terrible time. It's your job as a search engineer to say hey, this can't ship. Even though you wanted to tell your manager, even though you wanted to ship, everyone wants it to ship. It can't ship because of this reason and this metric is not working. We have to dig into why this isn't working. Even though we had this hypothesis, it didn't really show in the metric we wanted another metric went up and maybe you could launch with that metric but we don't understand why it went up so we should dig in more. All of these are kinds of it's a thought process that search engineers have to have that's very different from.

It sounds like engineer needs to have this mindset of experimentation and deep dive of a data Scientist and also has the business acumen can emphasize with a user.

Correct. Yeah. You really need to be able to do the deep dive on the data. If you can't understand that and you just want to ship stuff, you're not going to have a good time.

Right.

And it works both ways. Sometimes the engineers, we've sometimes had mistakes even on the team, really talented people who had at some point been working on search ranking and they get bored. They're like, I'm sometimes just looking at data that I can't understand. I'm not shipping anything. It feels like I'm not doing anything. And sometimes that's the job. Sometimes it is just pouring over data for days and just trying to figure it out and doesn't come easy. So yes, it's a specific role.

Yeah. And how do you evaluate people to see whether they have this them?

Well, we don't have, we're a startup, so we don't have very strict interview process rules. We have a general framework. Typically there's. We have assignments and tasks and stuff that we design to try to get some amount of general calibration across candidates. But a lot of it comes from, I mean, I can talk from my experience at least, I don't want to speak for the entire company, but in my experience I get a very good sense from just deep diving on an ambiguous problem with a candidate. And it's not always perfect, of course, but typically a good candidate will ask questions, not to you or not to solve the interview, but because they're curious and they will find holes in their thinking themselves and they'll be like, yeah, I like this idea, but it doesn't work because this doesn't catch this case. And that ability is really, really good. And I do understand that in an interview sometimes you think that you have to give the right answer so candidates can be taken off guard. But usually you get a good sense if you can make the candidate comfortable. The ability to deep dive into a problem and understand the whole breadth of it is. And then obviously specific data questions and.

Stuff like that are valuable. And in terms of engineering skills, do you do like lead code? Do you ask about their previous projects? How do you assess their capacity to writing production code?

All of them? I mean, we do an interview with past work. We do one interview which is unfortunately, it's algorithmic. So some questions are lead. Cody. I mean they're not, I hope they're not on lead code. Try to make sure that's not the case. But they are some form. One interview is just Algorithm design. One interview would be past work and deep diving into what you've done and what. And another big chunk is the assignment interview. I think that has the best correlation to your engineering skills and it's just telling you to solve a somewhat structured problem and just see whether you can deliver a project of code in two hours. So it's not like take home assignment or anything. It's like hey, you sit in front of us and develop this code in two hours and let's see where you go. Let's see how you architect the design of the code and whether you get it right and what trade offs you made and why.

Yeah, cool. And now with large language models and everything, AI, what do you think about the future of search?

It's fascinating. I mean even just this morning Microsoft announced that Bing would have a OpenAI integration with ChatGPT. I haven't seen what that product looks like yet. Google announced Bard and it's probably going to integrate that in search. You I first just want to lay of the land so you.com is a search smaller consumer search engine. They launched search result summarization with citations. So did Niva. So a bunch of players are in this space. But what is the future of search? I do. It's very hard to say. I'm really concerned about what this means for the ad revenue model. One reason why search works so well is because it's funded by ads.

Right?

When it comes to LLMs answering your questions, where do the ads come in? There's no click target anymore.

You can put it on the side.

But it won't get clicks and that's, that's gonna be gonna be a problem. I think these things, if you're, if you ever see the inside picture of Google, tiny, tiny changes in how they do ads make hundreds of millions of dollars and a change like this is a huge change to how the entire ads business works. So okay, that's one question. How can anyone fund this? The second question is, I mean even if Google can't fund it, Bing can say we will run less ads and we don't need to because we have other businesses and there's some interplay there. Other than that. Another thing that people underestimate is search is also a lot about distribution. So who has what your default search engine is, is typically the one you're going to use. So maybe nothing will change because Google already has all the defaults and it'll eventually integrate all of these things and maybe nothing will change. Overall my thought is there will definitely be LLMs will get cheaper, LLMs will get better, they'll get larger, they'll become more robust at handling a lot of things. There's going to be some important conversations to happen around copyright and privacy.

Yeah.

Because at the end of the day it's already happening. People are upset the LLMs are training on their custom data or a journalist is upset that I wrote this article and I don't even know when you are using this article to spit out a fact.

Right.

So there's a lot of these issues and it's very hard for me to predict the future. I mean, maybe the ideal version of the future is that LLMs always give your answer immediately when you want it.

Yeah.

I don't know.

And if you look at the speed of the change, I can't Even imagine after 12 months was going to be like, I think it's just going to be on my wildest imagination.

But I will, I will, I can, I can offer a slightly more pessimistic opinion here, but one thing that is an issue is all large language models and search is built on the web.

Yeah.

Now as we already know that the web is declining and it's A, it's very English, so it's really terrible in other languages typically and B, it's not increasing at a very fast rate. Even before this whole LLM thing, Google was concerned that TikTok had more searches for restaurants and stuff like that than Google. It was actively taking away from traffic. And then if you look at what LLMs train on, LLMs just train on text from the web mostly. So if the web market is anyways decreasing and now with large language models, there's even less incentive to create anything on the web because you're not even getting clicks now. Right. So the ecosystem is, is, is broken now in some sense because back in the day there is an incentive to contribute to the web because you might get some ad revenue, you might get some clicks, you might be able to drive your business or whatever. A lot of open platforms now, platforms are closed. There's no incentive. LLMs are summarizing all these answers. So LLMs can't answer what they don't know. And if all, if more and more information ceases to go to the web, then these models, no matter how good they are, aren't going to be able to answer things they don't know about.

Yeah.

And that is already a huge issue. I mean if you look at paper called Chinchilla by DeepMind and Chinchilla explores the concept of, or their summary is the only way to make large language models better is the bottleneck is training data. We need to give it more training data for this model to be better. And they've literally said, but we don't have any more. Like, we've done it all, we've thrown the entire web at it and that's not enough. So where does this training data come from? How does it stay up to date with new things? Where will information come from anymore? All good questions. I don't know.

Yeah. And also more and more bloggers now moving to substack, they publish their newsletters. So those things are also. Can be. Can be found just by general search. I think there's also.

And they're paywalled and stuff like that, so it's not even easy to crawl all of that.

Yeah. And what do you think about the AI power tools? How does it impact developer productivity?

Oh, it's huge. I mean it's already impacted my productivity so much. I use co pilot all the time. I think it does at least 40 to 50% of all of my code. And I don't like the narrative that, oh, this is going to replace engineers. Like anyone who says that has never done any engineering. These are tools to make engineering better and to make the solved problems easier to resolve. But fundamentally it can't produce solutions that don't exist before.

Yeah.

So yeah, if you're working on something that is just a reprint of something that existed before, yeah, it may replace that job. But if you're working on something new and innovative, then it has no hope of replacing that. But it does affect my productivity a lot. I do use it for not only Copilot, but a couple of interesting examples. One is I find myself much more, much better at starting side projects because one of the biggest barriers from. For starting a project was writing that first line of code.

Yeah.

And now I can just generate like the boilerplate first attempt, like this is a basic idea of what this is going to look like and then generate a little bit of this and generate a little bit of that and then I can start working on it and I'm already more likely. So that's one. I use it for regexes all the time now. So there's a lot of these little niche use cases where it's definitely affected my productivity a lot. Yeah, I think there I read somewhere that stack overflow views.

Yeah.

Went down like 10% since ChatGPT launched or something like. That's crazy. So it's clearly affecting everybody. People find answers to very obscure coding questions while trying to do this. And also it nails all the lead code. So it's really good at a lot of stuff.

Yeah, I agree. I don't think Copilot or ChatGPT will require will replace data scientists or engineers. But like you mentioned, it does increase your productivity by 40, 50% and without that, maybe say your team do need to hire a junior engineer. So I would say if what you do is basically taking the order from a senior engineer, do this specific product launch people give you 1, 2, 3, you put it into code, your job would probably replaceable. But like you said, if you are solving something new and innovative or something ambiguous, you still need to have that design thinking and your experience and creativity are still valuable.

And it's a simple thing that anyone can go and try even today is you try to generate a little bit of a complex program. With ChatGPT, the example I use a lot is generate a Python program that plays the game of Snake and there's going to, it's probably not going to work the first time and there's going to be like really subtle bugs.

Yeah.

So you still need somebody to find those bugs like these, these things aren't perfect. It's just trying to spit out patterns.

Yeah. And you, you mentioned you started some side project. What are some side projects you started?

Oh, I have a bunch. So I, I, I can tell you long story short, there's an exam in India called the IIT exam which everyone gives to go to engineering school in India. A million people give it every year and I happen to have the data set of all of the rankers from 2009 to 2016. That's in itself is not very useful to me. But what I wanted to do was track outcomes of people over that time to try to understand whether the top rankers in IIT over time have changed their decisions with professionally or in terms of like do they go to grad school more, do they go to America, do they go somewhere else, do they stay in India, what kind of jobs do they work? And I think that's really valuable for a lot of people in India who really care about that kind of information.

Yeah. So are you closely following the kind of tech news in India?

Yeah, I do, I do follow not so much tech news. I think I get a lot of my good India content from Twitter. I don't trust a lot of the tech news in India to be fair.

What do you think about the ecosystem? Startup ecosystem in India?

Well, I think there's a lot of positives like to to start out, India is clearly the future market. It's the only population that's really growing at that kind of rate. It is a place where so many people, I think per capita developers. There's no other country that has as many. And it's also so big, so many people are developers and so many people who are trying to move social classes now. So long term, bet, I think India is a huge market for any consumer product. However, in the short term, my understanding, my view is that most consumer startups in India don't make money. And the reason is because consumer behavior in India is very different than America. They don't have disposable income, people are extremely frugal and they have no brand loyalty. If you can get something that's 5% cheaper in another service, you're moving. Yeah, like, I mean, most Asians get that.

Like, they're very good at web search.

Yeah, yeah, yeah. They're, they're more, they're more tech adept. So it's. I don't want to spend and I will find a better deal if I can. Big, I'll go into discounts and it's really, really tough to get brand loyalty. And a lot of these models work on like a metric that people use in SaaS, for example, is LTV to CAC, which is boiling that down, is what is the lifetime value of this user for the amount I spent to acquire them. The problem is if you spend on advertising and all your users churn, then it doesn't work because you're gonna have to keep spending on ads every year to get people on your product. So a lot, like I would say this is probably, this is statistically true. At least 80% of companies, consumer companies in India will, are not profitable right now and they probably will never be. Yeah. And very few do work out and they're very nice. A lot of SAS companies in India work pretty well. But I. So there are pros and cons of the ecosystem. I think there is some more work to be done in core business models and what ideas really stick and work in India. A lot of good products, not a lot of great business models. That's what I think.

And also I had coworkers from India and they are frustrated about how long they have to wait to get a green card. So they want to move back to India eventually. Do you see a lot of people now going back to India, join startup or start companies?

Yes. So a lot of people do move back. Especially with the immigration situation now, it's quite hard to even Envision a life here. It's very, it's very difficult for a lot of people to even, even see a future here. That being said, I typically think that number is exaggerated. People always like to talk about Indians moving back more than Indians moving abroad. It's. It's much less sexy to say people are moving abroad, but yes, they are going back. A lot of people underestimate how hard that is. Sometimes though, going back after a long time here is difficult. You have to readjust to that life. You have to like reverse culture shock. You have to re. Remember all these things that you don't remember anymore and people are doing it. The salaries in India for tech are very, very high. So that might change because these startups might not work out, but they're very high. People live a very good life. The lifestyle is very different and a lot of people find it much more attractive than America. And to be honest, even if I think especially if you're above the age of 30, life in India is so much better than in America. You just, you're living like a king. So, yeah, I do see people going back. They start companies, they join pretty senior roles at existing companies and so on. So I hope the startup ecosystem enables them to keep doing that in the future.

Do you think you would move back to India at a certain point?

I've considered it. My personal decision making has at this point been, I am in America. Well, a, I'm culturally like more mixed than a lot of people. So anyone who watches this will know that my accent now is American, but I have an Indian accent. I can switch. I switch all the time. So. And I grew up half in the Bay Area and then half in India. So for me, culturally, it's a bit more of a mixed decision. But I do, like, I think I still agree that beyond the age of 30, especially if you have a family and kids, like, America sounds like a terrible life. Like, I don't want all these responsibilities like that I have to do myself. It just feels like a constant list of.

So what do you mean in India you don't have those responsibilities? Something else. Take care of your kids?

Well, there's a couple of a. You have your parents, right? So you have your parents that come.

And culturally they, they can. They like taking care of grandkids similar to China.

Yeah. And also like, you can just say, hey, my neighbors have kids and I'm going out tonight, you guys. Yeah. People are like, culturally much more close to each other. Right. You can't do that in America. Knock on your Random neighbor's house and.

Go like, hey, yeah, they might call the cops. It's a child abuse.

Correct. Correct. So that you have maids and drivers, so you don't have to drive, you don't have to clean, you don't have to cook. One of my really good friends moved from America to India recently with a kid. I think the kid was his primary motivation. And he literally says, every evening in the Bay Area used to be just, it's another day of work. Like, I end work, which is already pretty stressful, and I have, like a hundred things that I have to do.

Yeah.

And now in India, I can go home and just play with my kid for six hours straight. And he's like, this is the happiest I'll ever be. Like, there's nothing that can give me more joy than playing with my kid.

Yeah.

I can't understand that. I don't have kids. But it's a very strong review of India, so he's very happy.

Yeah. That's interesting. And you have over 30k followers on Twitter. What made you want to grow an audience there?

So the Twitter story is quite interesting. Initially, I had randomly. I made an account in 2011, and I would kind of shit post and randomly post things once in a while, and it never was a big deal. It would go into a black hole and no one would like it or comment on it. And I wouldn't do it that often, really. So then I think in, I want to say, 20, 18, 19, or maybe, yeah, 18, 19, I had a friend of mine who, I really respect his opinion, who, one day we were kind of grabbing drinks, having a good time, and he says, how many ideas that you have when we're drunk that you think are great ideas, but they're actually terrible? And I'm like, what do you mean? He says, they're terrible ideas. Like, never, never, ever tell anybody about these things. They'll think you're stupid. And then he says, what? Actually, let's test this. Like, I think his. His reasoning was, and he does this himself, is if you take the time to take a thought that you had, you might have a thought on an article, you might have a thought on, like, the tech industry.

Yeah.

Take some time to translate that idea and put it out in the public. You will realize that before you hit that send button, you will rethink everything.

Yeah.

You'll be like, oh, I don't know. Like, I don't know about this. I don't know if I'm ready for the world to see this. People are going to judge me. People might think I'm stupid. Is this really true? And what it really forces you to do is to ask yourself, is this really true?

Yeah.

And just that leap made me realize that I was thinking through ideas and getting much more clarity in my own thoughts just by posting on Twitter. And so slowly, over. Over time, I made that cadence more regular. I thought, okay, I want to do one idea a day, something that I care about, and then I will post about it.

Yeah.

And I think last year, 2022 is when it just kind of blew out. Blew up. Like, I never expected it to be that big. I was kind of just tweeting my thoughts.

Yeah.

I would be lying if I say that's all I do now. I mean, you get carried away with the followers now. I can't tweet random thoughts.

Yeah.

Because I'm. What will people think if I just say I had a thought about, this is why flowers grow this way. Like, no one cares about that now. I have to stick to my brand or stick to what I'm used to posting. So that's a side effect. That's not great. But I still enjoy it. I still post largely things I like. I put some more thought into them now. And yeah, that was a story.

Yeah. It's just like if I know I'm going to submit a code review, I'm going to write better code and just write something on the Jupyter notebook.

Yeah, that's a good analogy. Yeah.

And also I feel the same way. When I had smaller followers, I feel I can experiment, say something silly. But now, with now 200k followers, I have to think about, oh, is someone gonna get offended? So fun in a way that you get more engagement. But sometimes it's just someone can always twist what you're trying to say and then interpret something negative. I want to figure out a way where I can still be professional on LinkedIn, but also still have fun, not lose the mentality when I had a smaller following. But I know what you're saying. It's hard. You have more fear in your head than just putting out a thought to get an experiment, get a response from the world.

I have a question for you about that, if you don't mind. What what is. You don't have to answer this, but what is the worst way that somebody has twisted something you said or said something about you posted that that annoyed you?

Yeah. I'll give one example. So last year I posted about. I don't remember what exactly I posted, but basically you don't have to wait till you feel quite qualified before you apply for a job. And I talk about my experience. I said, if I wait for I'm 100% ready, I'll never get my job at Amazon. And I didn't say you should do the job when you are not qualified. I talk about. It's the feeling, it's the mentality.

Why you laughing? No, I can see. I, I, I, I. I'm on your side. And I mean, anyone with experience with, like, social media knows that these things like this, it's like just annoying, right?

The people. Yeah. Then they'll say, oh, do you mean I should fake my experience? I remember someone said, oh, does it mean if I believe I'm a doctor, I can just go there, give people surgery? I'm like, that's not what I said.

I think, I think there's a really good meme about this where I think this is very common on Twitter, where somebody says, I like honey. And then the response is, oh my God, this guy says he hates all foods.

Yeah.

And it's. You can interpret anything in a weird way. And I think it does happen.

Yeah.

You're a Data scientist. If 200, 000 people are seeing your thing, even if 0.1% are a little strange, they're going to say something about it.

So. Yeah.

Yeah.

And not just that. Someone I worked with, I worked, asked me about that. So they're like, what did you do for your interview? So I was a little bit nervous about that. I feel it affected how people see me at work and I don't know what my manager think of me. Did I get like HR's attention, the whole conversation? So it was kind of stressful. But looking back, I don't regret putting that information out there. And I think the only lesson I learned is just to be prepared.

Do you feel like you're better at being bold about what you post now or do you still think about it sometimes? Like, hey, I don't know if I should post this.

Yeah, I don't know. That's a good question. I think whenever I post some experiences, especially some failures, I definitely think about some kind of quote unquote backlash I got. So it makes me a little bit more careful. But again, I feel I don't want to live in this type of fear. I think I just want to be more clear about my intention sometime. I think, okay, am I sharing this because I want to get some attention or do I feel this story can help some people? If it's a Letter. I feel people can gain more insights about what is really like to be a senior data set or previous senior data science at Amazon. It's not about all the glamorous outside. People make mistakes, then I might still put it out there, but I might be more careful in how I word it. Add some caveats to my post.

So no, I can totally resonate with that because it's such a double edged sword. On one side it's like if you always show that you're perfect, people complain that social media is just a highlight reel and if you are vulnerable then it has all of these other consequences and it's, it's so hard to toe that, toe that line. I feel the same way. Like I struggle, like there's a lot of failures that I want to share but I struggle to share them because like people are just going to think I'm an idiot. So it is what it is.

Yeah. But I do feel more comfortable sharing after I left Amazon because when you work for a big company, there's a PR team. When you have a large following, a company doesn't need an employee to have a large following. They have their own platform. When you have a large following, you're more of a liability. So when I left, I still represent my current company in a way but I don't feel I'm watched by all the corporate communication team anymore. I don't have a perfect answer answer to that. What about you? Do you. How do you balance being authentic and also protect yourself?

Yeah. Well, thank you for sharing that. I'll tell you, mine is really, really simple. I think for most of the content, I mean just speaking about Twitter, for most of the content I post, my regular posts are not authentic because most of it is just factual. It's like it's not, it's authentic. It's not about me. Right. So I don't have to really care about what I say. This is maybe my opinion, but usually it's just factual. The more authenticity comes in the replies.

Yeah.

Because I know that it's not always foolproof, but I know that replies get a different audience and it's usually people who know you. And I think every social media has its own dynamics. But on Twitter, if it's people that follow you that see stuff that you write, even when it's incorrect, they're much more forgiving. The problem is when it blows up and people who don't know you see it.

Yeah.

And then they say all sorts of things and then so in replies, just the way the algorithm Works is replies never blow up. Only tweets blow up. Actual tweets. So in the actual tweets I try to be a little bit more conservative on what I say. And in the replies I sometimes let it rip.

Some spicy take from Dee Dee. He go through his reply session.

My replies have some, some actually.

Yeah, funny stuff. Cool. And what are some mistakes you made in your career that you feel comfortable sharing?

Well, there's, there's so, there's, there's so many. I don't regret all of these because it's hard to regret anything. But when I started out, out in my career, I made a conscious decision to go to Facebook. I don't think that was the right decision. All I optimized for was this is the least risk job that gives me a visa that pays a lot, which is what it was. It was great. But I realized in retrospect that if I had joined any of the other startups I had an offer from, I would have been, I don't want to say better off, but at least financially better off because every like I, I going through my. I'd done this exercise with a friend, but going through 2014, the list of companies was square. Palantir, I don't even remember some of them. But almost every company that I start up that I had an offer from has ten hundred extensized. So. So that is one regret. I was comparing offers on pure cash value and not future growth. I didn't really understand it. The second one is that I, I sometimes do regret. Not, I mean I not sometimes. That's a lie. I often regret. In your twenties you should take risk and not optimize for money early on because you're playing a stupid game with yourself and your friends where you're like, who's richer? You don't need the money because you're not doing anything with it. You're going on the same vacations, you drink the same things, you eat the same things, you go to the same places, everyone can afford all of that. But you play this weird game where you're trying to say who has more promotions and who has more money. Which is a stupid game because at that point it's the only time in your life where you can really just work 80 hour weeks and just grind on something that you're passionate about.

Yeah.

And that doesn't come back. And even though it's possible to start companies later, it's a lot harder. So I do regret. Like maybe I have excuses like Visa and this and that, but I could have gotten found a way around that and I know people who have and I do wish that right after college that's probably what should I should have done. Yeah just found some really smart friends and convinced them hey we should just for the next two years work on something and see where it goes. Not for money or reward. I think that would just be fun.

Yeah yeah.

So those two are the big ones that I, that I regret.

Thanks for sharing that. And how do you see your future career grow?

I mean it's, it's a hard question. I, I right now I really like what I do at Glean I a lot of the problems looking back at the 2019 post I wrote about Google Google a lot of those problems are not there for me anymore. You still have ambiguity with dealing with people in a business. You still have there's always going to be I don't want to use the word politics but there's always going to be people issues. There's always going to be engineering issues. Those things are just how life works. But I have autonomy. I have a lot more control about what can work and what can be done in this business and I really believe in what this product can build. So right now I'm not immediately don't have any plans and in the future we'll see. I, I, I do like I do in want to start something at some point and there's many ideas I'm sitting on and working on but it depends on when the other things in life have to work out for me to have two, three years of free time where I can just focus on that. Yeah and at least two, three years so that's, that's kind of where I'm what where I'm at.

Yeah. Cool. And right now what are something you're obsessed about like you're reading about.

Right now right now I mean these changes I like to read in general a lot and it's is a very this is the most non barrier area thing about me but I don't read I don't like reading non fiction I only like to read fiction because I think it's nicer but so I, I, I always like like to be on top of that stuff but in terms of what I'm reading that's more work related and what I care about Right now I really am diving deeper into Indian startups as like I've been tweeting about it for the last two weeks but just understanding Indian startup financials from the ground up and in general I think better understanding finances and Accounting. I did a minor in it a long time ago. But I really want to go back to the beginning basics of how do you look at a balance sheet and figure out and an income statement, figure out what's going on and understand what a business does. Just because I'm curious about it, I don't think it's really valuable for anything. And other than that, technically. Yeah, I mean I am. I follow all the research on search and large language models, but that's just a part of like my job.

Yeah. Yeah. I think understanding a balance sheet is pretty important. You're like basically kind of the job of a CFO if you want to build your own startup. I think it's important. Skills. So you like reading novels? Do you think you'll write a novel at some point?

Oh, I would love to write a novel at some point.

What would you write about?

But I really like the idea of writing a very good horror novel.

Okay.

It's a very hard genre to write about. But like, as an avid reader, like, it's crazy to me that you can read and get scared that like words can do that to you.

Yeah.

And so I. I just love the format and the medium of reading. Even though movies and stuff are great and I love movies, but there's something magical about inciting an emotion by structuring words in a certain way. So yeah, I would probably like to write something horror, multi genre, but mostly horror on. I don't know anything else more than that. I have a couple of ideas, but I don't know if I'm comfortable talking about them right now.

We're not going to scare our listeners, but maybe one day when you publish a novel, you can let us know.

Yeah.

It was a great conversation with you. I learned a lot about the search space and your career stories. So for folks who want to follow your journey, where can they find you on the Internet?

Well, you can find me on my. My Twitter is probably the best and most active place. So it's my full first name and last name you can probably get by searching for Didi Das. DD is D E E D Y, but that's not my handle and my website is my first name, last name, dot com.

Do you want to say your first name, last name? Yes.

Okay. My first name is pronounced debargo, which is why it's very hard for everyone to say it. And it's spelled D E B A R G H Y A and my last name is D A S and it's first name, last name, dot com and Twitter is firstnamelastname. So that's where you can find me on the Internet. I'm probably tweeting every day. Find the fun ones in the replies.

Cool. Thank you.

Thank you so much, Daliana. This was a great conversation.