[AI and the Modern Data Stack] #181 Why the Future of AI in Data will be Weird with Benn Stancil

And once you have some new kind of foundational piece of technology, a bunch of new stuff happens with it that you wouldn't really have been able to imagine even existing before. Like, it's, it's, you know, there's, there's, it's very hard to explain if you, if you were pre iPhone, like what people are doing with TikTok, like, like the sort of, that stuff is like, doesn't really even make sense until you like imagine, oh, you have these phones, you can like do all these sorts of crazy things with them. You know, you have like all these sort of location services built into them. There's all these other like capabilities, like you have to kind of have the phone first before you can even imagine like a thing like, like TikTok or a thing like Instagram or whatever. And I think AI is to some extent that we're like, whatever happens to it, it's going to be, the people see it and they're like, oh great, this is a chatbot. Okay, we're going to go build a bunch of chat bots. It's like not, probably not like that's, that's obviously a first thing. You know, when you first build a phone, a bunch of people build sort of the easy extension of what the world was like before. But once you start to actually play with it, you realize, oh, this thing can do all these sort of unexpected and weird things or sort of have our mind is opened up to a bunch of ideas that we wouldn't have really thought of until we got to this point.

Ben Stensel, it's great to have you on the show.

Good to be here. Thanks for having me.

Thanks for coming on. So you are the co founder and chief technology officer of Mode, which was acquired by ThoughtSpot this year. You also have an incredible substack that I highly recommend listeners subscribe to. A lot of thought provoking takes on the data space. We also have a bit of a shared history that we both tried our hand at political science. You were a fellow at the Carnegie Endowment for international peace in D.C. and I was a research assistant at the Carnegie Middle east center in Beirut. So maybe before we deep dive into the meat of today's chat, walk us through the journey from starting at Carnegie to ending up founding Mode and getting into the data space.

So I graduated from college in 2009, which is relevant because nobody hired for any jobs. Like 2009 was when everybody was getting fired, people were getting offers and then getting them revoked before they got there. And so like the one place that was hiring places people was D.C. this was basically the only place you could find jobs. So my background was in, was in economics and so I ended up working at Carnegie. Carnegie was like a think tank. For those who aren't familiar, think tanks are basically like a half academic, half policy organization where you do kind of research, but it's aimed at being like, how do you affect policy in sort of more specific ways. It's not academic research as much as it's saying, hey, we should enact this policy, this is what we think of this bill or whatever. And so I ended up there partly out of like it was related to what I was doing, but it was also, you know, the job you could find in 2009. And so it's like very interesting work in that it's, you're thinking about big problems. The way that you do the analysis is interesting. I was basically like looking at economic data and trying to make recommendations about what various policymakers should do. However, nobody listens to you. You are 20 something year old, writing policy briefs that are then sent to Congress and Congress does not care. You are eight steps away from anybody doing anything with this thing. And so like academically interesting but not something it felt like this is, I'm just, you know, yelling into the void here. And so I ended up, you know, as, as you are familiar, think tanks are sort of dead end jobs in some ways. Like you typically are their stepping stones to grad school or some other like government job. And then you come back to the think tank when you're on the cusp of retirement. And so I started looking for other things. Had a couple of friends who had moved out to San Francisco, liked the jobs they had there and then found a job at a company called Yammer that was just like, it was like a data analyst job. It was, you know, look at data, make recommendations for the company in some ways very similar to what the think tank job was, which was like, here's an ambiguous problem, go do a bunch of analysis, made a recommendation. Except instead of the recommendation being to Ben Bernanke, the recommendation was to a PM that you sat next to. So less impactful in the sense that you weren't trying to, you know, look at policies that could change the world. But the person who you're talking to was like, would listen to you and you were, you weren't like a free consultant that they never asked for that just decided to make recommendations. You were someone who was there to like try to help them. And so same, same structure of job Much more felt like you were actually doing something that somebody was, somebody cared about.

Yeah, I can attest to that. It's definitely intellectually very stimulating, but the impact is quite limited in the think tank space. And maybe following up after your experience in Yammer, or maybe give us the why behind founding Mode. What was the problem you were trying to solve for data teams when you first had the idea for Mode?

Yeah, so I joined this team at Yammer. We were the sort of data team that our job was not to build dashboards and just reports, but we also weren't the kind of like data science, do hard math in a corner and come out with a model that predicts the world type of stuff. Our job was basically the marketing team would come to us and be like, we're holding another event. Do we hold it in San Francisco or New York, like, help us figure out the right place to do it? And you do some stuff like, well, we have customers here, prospects here, this is where we think we should do it. Or like product team would be trying to make a decision about we ship feature A or B or those sorts of things. And so we essentially needed a tool that was technical enough for us as a data team to do a bunch of exploratory analysis that we were working with data and Vertica at this point. So it was like a database writing a bunch of SQL queries, doing that kind of stuff. But we also had to collaborate very closely with the people who are next to us and they weren't technical. And so we just needed to be able to share like reports and stuff for them to be able to do their work as well. And so we ended up building this internal tool inside of Yammer that was basically like a SQL query editor in a browser where you could run queries and then send them off via URL to other people who could then just like look at charts of the results. And it was ended up being like this really impactful tool inside of Yammer and then inside of actually Microsoft after, after the company got bought because it allowed us to work as like the technical data team in the way that we wanted to work. But we could very easily, like collaborate with the folks we were working with because we weren't working in like a jupyter or an RStudio or something that we couldn't send to them. We were working on a thing that we could very easily, like, know there was a back and forth there that was easy, where both of us kind of had a, a view of the data and the Results that, like, fit kind of our, our technical abilities. And so actually, after we, we built this, we started to realize that one, this was an internal tool that was pretty successful inside of Yammer. It was successful inside of Microsoft. A bunch of other companies around Silicon Valley had actually built similar things for their, their data team. So Facebook, Airbnb, Spotify, Pay, Pinterest, Uber, all had these kind of like, query tools in a browser type of thing. And so essentially me and the other two folks started it, were kind of like, hey, if everybody's building this thing, if everybody thinks it's like, been successful inside of a company like Microsoft, like, maybe there's a market for it where somebody should just build it and, you know, turn it into a product where everybody's going to have. And so that was basically the impetus of the idea, was the success of this internal tool that we had built and thinking that, you know, as databases become cheaper, as data teams become more of a thing, that this is the type of way that they're going to want to interact with, with their kind of business stakeholders. And so can we build a tool that leans into that kind of workflow?

That's great. And in a lot of ways, maybe this is a bit of a wide question. Your experience at Yammer, which was at the time quite an advanced organization from a data perspective for the time, led to the founding of a tool like Mode. How have you seen maybe the Altix space evolve since that time? What do you think are kind of the main highlights of evolution that you've seen in Space?

I mean, there's a lot. So we started when we started Mode. One of the biggest objections that we ran into was it was a cloud tool. Like, it was a SaaS product. And there were a lot of people who were like, well, data SaaS stuff. I don't know that that makes any sense. Like, we're not going to buy that. This was one of the hardest things we had to sell around in the very early days was, was just general nervousness about the cloud, specifically as it relates to data like people. Salesforce was obviously like, already a very big thing at that point. Cloud software was a thing. But, like, data in the cloud was kind of like, I don't know about that. That changed a lot. Redshift was probably the first big, like, hey, this is a thing that might actually work. Snowflake, I think, was the, you know, and this was much later, but like, the Snowflake IPO was somewhat of a big moment in this because it was like a bunch of People who, you know, like the, the people who pay attention to things with very big dollar signs in front of them started to realize like, oh, cloud data is like clearly potentially a very, very, very big business. That changed, but it changed kind of gradually over time. The other thing that, that also was very different is when we first started pitching mode. One of the biggest objections we got from like VCs and VCs don't. Not that VCs know everything, but like VCs are a decent reflection of kind of like conventional wisdom of what people think about these sorts of markets was who's going to use this thing? Like this is a tool that writes SQL. Who writes SQL anymore? Like isn't that over? I thought we, we had Hadoop and like why does SQL matter anymore? And there's only a hand. Like these are a bunch of handful of like nerdy IT people in the back corner that do it. What is this all about? And I think one, the Hadoop era didn't really stick and so moved away from that. But two things like the modern data stack and a lot of the philosophies in that of it being a very SQL heavy thing of tools like DBT and again the popularity of something like Snowflake, which is obviously a very SQL heavy database that became a much more like oh, there are a lot of people whose job it is to build data infrastructure that is primarily SQL based. And so the notion of this like modern data team, that there are people who are writing SQL, they're trying to answer questions, they're not just building dashboards and reports. They're not kind of like the capital D data scientists. Their job is to sort of like figure out what an organization does with its data and make it useful. People are kind of skeptical of that idea. 2013, 2014. Now it's like that would became much more of like this is the way that this stuff gets done.

If anything, the analyst has become kind of the bread and butter of a successful data team. And today in driving the data driven, the data informed organization, to quote your substack. Now actually we can talk about the modern data stack quite a lot. But what I'll talk to you about today is especially the potential impact of AI on the analytics workflow. This is something I've seen you write quite a lot about in your substack and I find a lot of interesting takes here. And you know, one thing that we chatted about behind the scenes when we were talking about this episode is that resonated with me a lot is that you talked about AI assisted analytics workflow in the future being weird. And I think the word weird does a better job than most other adjectives describing the potential world we're heading into. So maybe you've set the stage. What do you mean when you say the future of AI and data will be weird?

Well, I said two days. I get one. One is generally to me the way, like, tech, and this is not a novel thought necessarily, but the way that like, technology and stuff changes, like big technological shifts change how people do things. It's like teleportation in some ways that you, we don't like, sort of walk slowly towards it where we can see what's coming and it's like, oh, we see it in the distance and we get closer to it. It's like a big thing comes out and you sort of get teleported to a new place where you have no idea what's on the other side of that thing until you get there. Like, if you like snap your fingers, like, guess what, you're going to teleport somewhere. You're like, I have no idea where I could end up anywhere. Like, it's impossible to imagine the next five steps when you do that. And I think there's like a version of that where these big changes happen. And once you have some new kind of foundational piece of technology, a bunch of new stuff happens with it that you wouldn't really have been able to imagine even existing before. Like, it's, you know, there's, there's, it's very hard to explain if you, if you were pre iPhone, like what people are doing with TikTok, like, like the, the, the sort of, that stuff, it's like, doesn't really even make sense until you like, imagine, oh, you have these phones, you can like do all these sorts of crazy things with them. You know, you have like all these sort of location services built into them. There's all these other like, capabilities. Like you have to kind of have the phone first before you can even imagine like a thing like, like TikTok or a thing like Instagram or whatever. And I think AI is to some extent that where like, whatever happens to it, it's going to be. The people see it and they're like, oh great, this is a chatbot. Okay, we're going to go build a bunch of chatbots. It's like, nah, probably not. Like, that's obviously a first thing. When you first build a phone, a bunch of people build sort of the easy extension of what the world was like before. But once you start to actually play with it, you realize, oh, this thing can do all these sort of unexpected and weird things or sort of have our mind is opened up to a bunch of ideas that we wouldn't have really thought of until we got to this point. So generally to me that's, that's like it's going to be weird because it's, it's a, it's a big foundational shift that we can't sort of just extrapolate the previous what we've been doing. It's like, it's like a discontinuity. The other thing to me about them specifically is AI and LLM specifically are a very weird kind of technology where like they're very human in that a lot of the things that humans do that like they are pretty creative, they aren't very good at taking explicit directions and doing exactly what you want them to. They do unexpected things. They like, you can treat them almost like humans where like consistently, to me, the thing that is surprising is when you want them to do something better, you do the same thing you would with a person where you like, you just give it more information, you kind of explain it more stuff, you ask it for more specific results. Like you tell it to give it feedback and it kind of learns. Like there's a bunch of weird stuff in that. That's like a pretty weird. It's not a computer. It's a computer obviously, but it's not like a computer that thinks the way that, that we have always thought of computers thinking. And so to me, like, what happens to that? I have no idea. Like there was this, this was the Ilya, the, the. Yeah.

Chief scientist.

He is currently now at OpenAI.

Yeah.

Had a thing about like people complaining about, about hallucination in LLMs and he's kind of like, look, hallucination is the point. Like hallucination is. It's not hallucination, it's creativity. And that the problem is not the problem. Like he was comparing to search and he was saying the problem with search is it has a. It does, it has no creativity. Like there is. You search for something, you're hearing the same thing all the time. Like it is not a creative thing. But LLMs are create like that. That is sort of the benefit of it. And I don't think we've grasped that yet. I think we've like view them as how do we make them perfect little robots that do exactly the task we want. And it's like they're not really that they're kind of this weird creative robot. And that's kind of a crazy thing to think about. But like what happens that I don't know, but we're not used to, I think having a tool that is more creative than we are in a lot of ways, but is not good at following directions.

Yeah. And there's a lot to unpack here. And you mentioned at the beginning is that, you know, you compared large language models in AI to the, to the iPhone and that, you know, it's hard to explain TikTok pre iPhone. And I completely agree that notion that AI will usher in an ecosystem of applications and tools that we didn't think possible. And you know, in a lot of ways chatbots today are those first waves of apps that we've seen after the ipod touch and the iPhone was released. Do you remember like for example, the beer app where you could. Yeah, like we're seeing kind of the, kind of the first wave of applications of large language models today. And in the future we'll see, I think a lot more robust, a more robust ecosystem of tools emerge. But let's ground this discussion maybe right now, given the talking about applications to the capabilities of large language models and coding workflows and analytics workflows. Right. We've seen a lot of promise actually when it comes to coding workflows. Things like GitHub, Copilot, the ChatGPT data analysis formerly known as code interpreter. So maybe anchoring our conversation first in the present capabilities. How do you view the current capabilities of large language models and being useful in coding or data workflows? Where are they effective and where are their limits?

There are, I would say two things about that I think. One is they are not effective at all as the wrapper around ChatGPT does not work. I think that there's, you know, There are the 1000 YC startups that are like, it's your data analyst assistant. You ask it a business question and magically you get an answer like what was my revenue in California last quarter? And you know, it writes a SQL query for you or whatever. This is a pretty common thing. People have built versions of these. They have basic sort of like prompt engineered ChatGPT and fed it a little bit of context about your data and like hope it writes a good SQL query. And it, those are the, oh geez. Those are the sorts of things where like it does a good enough job to convince you that this might work, but it really doesn't work. So it, it's like those things are, you can do it on top of these very toy data sets, you can do it on top of the schema that's got four tables and 12 columns total and they're all perfectly named. And you're asking it like questions that you can kind of figure out from the schema and it will do a thing that is convincing enough to be like, ah, yes, this might actually work. But then when you start to use it for real, it does not work in actual business context at all. Um, it just is like, and especially when you're trying to get an answer that's very, very like the CEO asked, what was our revenue in California? You can't be like, well here's our best guess. Like that doesn't really make sense. So I think in those cases it is not, it is not good. You can do a bunch of work around it that is if, if you add a, a ton of work underneath it, like you can kind of coerce it into doing the right thing if you prompt it with a bunch of the right stuff. If you do a bunch of work to figure out when somebody asks what is our revenue in California, what does that question really mean? What is the schema that we should be feeding it? Like there is a lot of underlying infrastructure there that you can do that improves that a lot. Even there though, I think it's tough, it's hard to go from question to SQL query kind of no matter how much context you give them. Maybe that'll get better, but currently that's a pretty heavy lift. The thing that I think it does a better job of is, and this is so, so we have a, we're building sort of a tool inside a mode that is basically this, that we're starting to roll out to some customers. The aim is much more of like a co pilot style of thing where instead of asking a business question, you are basically asking for help to do the like it is for the technical person asking for help to do the thing that they want to do. Where you as a, as an analyst get asked, what is our revenue in California? You roughly know the SQL query you have to write to do that, but you don't want to do all the tedious work of like, how exactly do I define this? Join. What are these like various functions in SQL that I want to do? I got to bucket it by something. I don't want to write the giant cake statement that does that. Like you are essentially using mail to go from pseudocode to finished query as opposed to from business question to like query that it, that it does because There's a bunch of things where it's like, if you go from business question to query, it writes queries that are kind of weird, but like, even if they're right, they're kind of hard to parse. And so if you're like, I don't fully trust it, I have to go through and read it and like code review the thing that the machine wrote without understanding the way that it did it. And it does things in ways that are a little bit like off kilter from the way that an analyst would actually do it. It's a very painful thing to be like, all right, Great, here's a 50 line query that I have to figure out what this thing is trying to do and if it's right. And so I think it's actually much more effective to be like, let me tell you roughly the query I want to write where I just like dump out a bunch of like, do this, join this, join this, join this, like, do it in sort of like the way I would just yell at you if I wanted someone to write it. And then they like, it just takes all that and does the like tedious work of turning it into a query that to me is like a much more effective way that these things get used today than the like, business question to query.

And when you're looking at, you know, the. Now we're looking at the current state of large language models for coding workflows, and I agree with a lot of the limitations that you lay out. I think providing that really deep context over the company's data and translating from business questions to SQL queries is going to be much more challenging than we expect it to be. But if you project a lot of the current capabilities into the future, how do you see these capabilities ushering in the weirdness that we've described earlier? What do you think will be weird about potential analytics workflows in the future?

Two things, I think. One is that, that even if they're not, I think, I think it'll be a long way. And this is, I don't know, these things move pretty quick. And so I, in the early days of COVID I was always like, this thing will be over in two weeks. And then I'm like, it'll be over in a few months. Always basically like a month behind what was actually happening. And I feel kind of the same way about this stuff. I'm like, it definitely can't do this. And then a month later it's like, no, it actually did a pretty good job of that. So with all of this, with I have, you know, no idea how to pick future, it seems like they're gonna struggle for a while with just like writing a really good query out of the box. And part of that is also, again, I think like a little bit of a misuse of what they're good at, where it's like, they are kind of creative. They aren't great at taking directions. Exactly. Us using them to do things that are very much taking directions and not being creative is like a little bit of a misuse of technology. But I don't think they're gonna like writing a, you know, actual asking complex business questions to generating exactly the perfect query from there. The two things I do think that they will be good at that are the things that will start to get weird is if they get good enough, like understanding a question, I think we can start to build infrastructure that is designed to help them answer those questions that doesn't quite. It's like right now there's a lot of people being like, let's take the documentation that we wrote for analysts and stuff it into the context of this thing. Let's take the queries that analysts wrote and stuff into the context of context of this thing. It's all like taking things that were designed for today and trying to train it for, like, trying to help the model understand stuff. And it's like, actually we may end up realizing that if we build things specifically for it, it's going to get a lot better. So part of that is this is something like ThoughtSpot, the company that Mo got acquired by, has been thinking about this stuff a lot. They have basically built an underlying semantic model that is designed for an LLM to. To use as opposed to design for a human to use that kind of concept. To me, you could extend in a lot of different directions where it's like the system, instead of it being all designed for humans, where we stick an LLM on top. It's like these things have some weird properties and some things that are slightly different than humans. What if we cater to those weird properties instead of catering to people? The second thing I think that is potentially weird is the way to me that the times I've tried to use them to do analysis, they're not great at writing a query, but they're pretty good. When you're like, let me describe to you a situation. This number's up, this number's down. These are these things. What should I look at next? Come up with some ideas. They give you some pretty good ideas of like, here are 10 hypotheses about why this could be the case. And you're like, how do I explore that? And they go and give you some ways to like questions you could ask. Like it's not doing the work for you, it's doing like what analysts say is the fun part. It's doing the part of like coming up with the creative stuff. And, and to me that's a little bit of what this actually could do is it's like a lot of people say, oh LLMs, they'll automate the parts of the job we don't like and we'll get to focus on the fun part. I'm not sure they're not going to just do the fun part. Like I'm not sure, become the query writing robot that is responding to like the prompts that it's telling us to do about like you should try this or try that because that's actually what they're a lot better at. And so I don't know, there is a sort of weirdness there to me where the job that we want is like sitting there and thinking about problems and solving puzzles is like, they might be better at that than we are.

And then when you're talking here about the, this is very fascinating. You're talking about the potential uses of large language models for the actual creative aspects of the data job. Right. And even when you mentioned kind of that first element of providing the large language model infrastructure to be able to ask better questions about the organization's data, combined with that ability to come up with creative problem solving ability, I think could be radical, radical in the future. Let's maybe focus here on what that world would look like in a lot of ways. The current interface we have with Data think tables, BI tools, IDEs are not built for a world where machines can potentially reason about problem solving, code different solutions. How do you view the future of the data interface there then? And what do you think our interaction as analysts with data look like?

I'm not sure. Like, I don't know. I think it's possible that, that you know, we, they're not good enough at this yet and they're, it's a little bit of a pain. Like, honestly, I think that in some ways, like one of the things that's also hard here is you have to be very particular about what you tell it to do. Like when you're trying to say like, hey, I want you to help with a thing, you end up sort of being like, and here's this, like it's it's work to tell it what you needed to do. And so I think that there's. There's some stuff there where it's like, still not a great process, but.

I.

Think that one, like we may end up just spending a lot of time. It's not prompt engineering, like, I think prompt engineering is the analogy I would use on this is back when like search engines first came out. I remember like in middle school, people were like, you need to learn how to do various Boolean search stuff in Google. Like, you need to learn how to do like include this word minus this word. And there's like a little bit of like a query language in there that was like, this is the future is this sort of language. And like that was right in the sense that Googling was the future. And it's a very, like, if you can Google things, that's a, that's a valuable skill. But it wasn't right in the sense that like, that's not how you learned how to. It wasn't you learned this like, particular language for Googling it. It was you, like, you kind of learned how the system worked.

You learned.

Learn the ways around it and how to sort of make it do the things you wanted it to do. And I, I could imagine there being the same thing that happens here where like prompt engineering, which is this kind of like more structured. How do I use LangChain or whatever became the first version. But to me that's like the query language in Google thing. Really what it is is it's like you kind of learn the tendencies of the models. You kind of learn like how to coerce it into doing the thing you want it to do. And so I can see there certainly being a. The analyst job is partly like combining the skills that you have is like reasoning through a business problem, but also coercing like the LLM that is probably better at some of these things than you are into helping you do what you want to do. And so it's like figuring out the way to sort of operate the machine where like, all right, I need to describe the problem to it. I need to help it, like, coach me on some stuff. It'll come up with some ideas. There's this. It just came out yesterday, but it's a book I want to read. There's a guy who wrote. I don't remember his name, Seth Davidson, David with something. He wrote a book called Everybody Lies. It was about like the things that people admit to Google because they're like looking for stuff and they think nobody's looking Seth.

Stephen Davidowitz.

Yeah, yeah, yeah. But he just wrote a book that was about the NBA and the, the, the gimmick of the book was he wrote it in 30 days with all AI where he essentially did a bunch of like analysis and stuff on it and it was all like, he basically just like chat GPT a book. And I'm interested in reading that because I think like that probably is how this actually works. Is like he produced a hundred page book full of analysis in 30 days. Is it good? I don't know. But like there is some skill in that that I'm sure that he developed that isn't prompt engineering. It's not strictly analysis. It's like figuring out how to use this thing to be really, really effective and fast at doing analysis. And I think that's, that's probably like what this really looks like is you learn how to do that sort of stuff. I don't know. What does that mean? I don't know. But like how would you have told someone in 1990, you're going to have to learn how to be a really effective Googler if you want to be a functioning human being in the world. I don't, I don't really know what you'd even say but, but you're like, look, there's going to be this kind of weird technology, it's going to do all this crazy stuff, but you're going to have to be really good at it otherwise you're going to be like incapable of functioning in modern society. And it seems sort of like that's what this is. It's not, go take a bunch of courses on the ins and outs of LLMs. It's kind of like you're just going to have to learn how to ride the bike.

Yeah. It's hard to predict here and especially when we think about how the skillset of the modern data analyst, data practitioner will evolve. You know, if we take the assumptions that we're discussing that more and more LLMs will be able to assist in creative problem solving and in actually creating a lot of the code that goes into solution. What do you think will be the main skill sets data practitioners will need to evolve and kind of develop in the future. Like what will be, let's say grad school in 2030 or a data degree will look like?

I mean, I think grad school will be the same. I think it'll be going to be a bunch of learn some matlab and some like stats. I don't know, Will it be useful different question, but it'll be the same. I mean, I think like they. The skills that will make people good. I mean honestly, it's probably like the skills that make people good now, which is. Well, okay, actually I think it's the skills that make people good now, but. But maybe less. Maybe like in a different way.

So.

So there's this interesting thing. This is a, this is like a more, I don't know, delving into philosophy that I can pretend I'd understand things about, but I don't. One of the things to me that is tricky about LLMs is they're able to make pretty convincing arguments and do pretty convincing like analysis like this. I imagine this NBA book has convincing charts in it and like logical arguments and data analysis are hard to poke holes in. Like, that's the point. This sort of like debate club. Like if you, you argue with someone who's really good at debate, they're going to kind of defeat you at debate because not because their argument is better or like they're right, but just because like they know how to play the game and LLMs know how to play the game. They know how to say stuff that's convincing. And I think that there is something that's like tricky about this where when you can just say to an LLM, make a good argument for this point or like do analysis on this thing, it will give you something that is relatively convincing and there is no like foundational truth to that thing. Like if I have to come up with an argument that is convincing about some analysis, I'm not that smart. And so like I don't find good arguments in it unless they're relatively easy to find. It's hard for me to go and like find the totally obscure thing that proves my point. I like I, the. The reason I was able to make a good argument is because their argument, good argument was like fairly close to the surface. But if you're like this all powerful super. Not that all powerful supercomputers, but like in effect if they're all powerful supercomputers that can always find the, the connect the dots in this thing that tells a compelling story, everything will look compelling and we're not that good at figuring out if it's not. And so I think like actually in a lot of ways the job of an analyst will be what the job of probably this, the Seth was in writing this book, which is this thing is going to give you a thousand things that all sound pretty convincing. Which ones are real. Like what's, what's what actually is truthful in this? Like, it's, you know, if I. If I ask it to be like, give me a very convincing. You know, this is a different example. But say, like, give me a very convincing analysis on why drinking a glass of wine every day is good for you. It'll give me something very convincing. And I say, give me a very convincing argument on why drinking a glass of wine every day is bad for you. It'll give me a very convincing argument. What do I do? Like, my. My job is to, like, read these things and actually be like, this one seems better. And that's, like, pretty hard to do. And so I think actually a lot of the job of this could be we have a thing that will tell you whatever you want. It will give you a convincing argument for any point you want to make. My job is like, now try to figure out which one's better. Like, it's to sort of moderate the debate, you know, I don't know.

Do you think that makes the data job better?

I think it makes it not better for the. There's some people who, like, I want to put on my headphones, I want to, like, sit in a room and. And, you know, have my three monitors and, like, write code. Like, I want to be an engineer. I like that mindset of, like, solve the puzzle. I don't think it's better for that because I think you become less of a builder. I think you become less of, like, I'm. I'm creating a system. I think you become more of a. I gotta reason about a bunch of stuff. I gotta sit here and just like, think about which one makes sense and kind of, like, poke holes in it. And there are analysts. Some analysts love that. Some analysts love being the person who's like, let me debunk every single thing you say, because that's, like, fun. So I think in some ways it's that. But if you are the analyst that got into it because you kind of actually have the mindset of an engineer, I think it's kind of less of that.

One thing that we haven't yet discussed on the potential uses or kind of the weird avenues large language models open up when working with data is something that you mentioned in your most recent article, Average Text. You describe a hypothetical world in which a bar owner summarizes user interviews grouped by age group using a simple query, where here we use a fictional SQL function called average text. This, in my opinion, illustrates a really, really. Well, illustrates. Sorry, a potential for large language models and surfacing really detailed Granular insights from unstructured data that will fundamentally change how organizations make decisions. If you can at a fly summarize user interviews grouped by demographics or grouped by product, for example, to try to understand how a particular group feels about your product, I think that gets you much closer to reality than just behavioral exhaust. So walk us through how you view AI will fundamentally change how we work with this type of data and what you think the opportunity here is.

This is a little bit of a. Maybe this works. I don't know. Like I, we have this. To me, a lot of the data industry has gotten built up around and like data industry sort of be bigger than the data industrial complex. We'll say like the, it's. It's partly like data companies, but it's also like this mindset. Like data is the truth. This is like, you know, that kind of stuff. I think a lot of that has been built up because data, data is the it. Like it feels scientific and in some places, like if you are trying to, you know, make a vaccine for Covid, great, please do the science. But if you're like a product manager trying to make a decision about something, you actually just like want to understand what customers think. And, and data is an artifact of that. It is not what they think it is. It is, you know, it's a proxy. Yeah. And it's kind of a lousy proxy, to be honest. Like it is in some ways it is. But it's like, okay, we see what people do, but like we interpret a bunch of intention from that. But really you're still just trying to get at the int. Like you really just want to know like, what's in these people's heads. Unstructured data, customer feedback, you know, interviews, user research, basically is what's in people's heads. And like, you can't just take them at face. You can't just be like, this is what this person says, therefore we have to do exactly what they say. Like, obviously you have to, you know, sort of still think about what it is, but there's, there is so much more richness in that than the kind of exhaust that comes out of their behavior. And so, but I don't think we think we like view the unstructured one as kind of the like soft, squishy stuff that's not as valuable because we don't have kind of like scientific implements around it. And so I think that like, not that LLMs give you the scientific implements exactly, but they give you ways to look at unstructured data at scale Potentially where it's not like, great, we did an interview with 10 people. How do we like extrapolate this but also add all of the caveats that it was 10 people and who knows? And all this stuff. It's like, actually, what if you can look at a thousand customer, look at 10,000 and sort of like summarize them and like do that in a way that, that, that scales. It starts to be like, you can look at a data set that is as rich as what you would get in this unstructured stuff, but with some of the like, properties of looking at things at scale. Some of the like, okay, the statistical significance isn't exactly the right thing, but like some rough concept of, hey, a bunch of people said this, it's probably real. We don't have to just be like, well, we talked to 10 people and eight of them said it, so that's probably enough. It's like, no, we can look at kind of the totality of how people feel and stuff like that. And so if you can do that, there's a lot more richness, I think there than there is in whether or not somebody really visited a pricing page. Like, like right now, if you want to understand is our product priced correctly? The way that we'll do it is like, look at people who visit the pricing page and what do they click on and like try to do some sort of demographic things of like the people who fit this, clicked on this and like, try to infer a bunch of things from that. And to some extent that's not bad. But what would be better is like, what if everybody visits the pricing page? They, you know, you said, as they're leaving the pricing page, can you give us a 15 second audio clip of how you feel about our pricing and we'll send you a $5 gift car gift certificate to Starbucks immediately for 15 seconds. I would do that. I'd click the button and be like, I think it's too expensive. And now if you have all that information, like, that's going to tell you way more than you trying to infer a bunch of weird stuff from like user behavior. And so I don't know, like, does it work? Maybe, maybe not. But I think if we start to think about like unstructured data as being the information in there being more accessible and we start to try to collect more of feels like there's a lot more sort of informed decisions you can make that way than you would off of kind of again, trying to parse your way through how people feel through this Kind of like the footsteps that they leave behind.

Yeah, that's fascinating. And you could probably use an NLM system of some form to scale the amount of interviews that you can make in a short period of time as well to get. So it's interesting once you open up the imagination to how these systems can fit into the data collection workflow as well. So now we're talking about a lot of the long term stuff and kind of what we expect to be weird in the future. But let's also focus on the short term thoughtspot which you are part of now. You mentioned the semantic model of working with large language model in my opinion has a great elegant solution of embedding AI into the analytics workflow. Maybe walk me through how you view the short term of AI analytics and data.

I don't think that much changes. I think people will in the short, immediate term. It's like people trying to figure out what works. A lot of experimentation, a lot of things that are kind of going to be busts. Honestly. Thoughts about, thoughts about. Have been doing this for a long time and one of the things that I appreciated about it when we, when we started talking to them before the acquisition was they have been in the trenches of this stuff for a while. Like they had been building things that were sort of AI powered well before the LLM hype. And so they had seen the things that work and don't work and sort of had been through the like what if a chatbot wrote a SQL query and they're like oh God, this is not going to be good. And so I think there will be a lot of like experimentation with chatbots to do that kind of stuff. And every, you see this now like every data tool has some flavor of product that's like right question and it'll magically write a single query. And I think we'll get kind of inundated with some bad versions of this. Just because it's a two week project to build a wrapper around OpenAI or Bard or whatever. It's a nine month project to make that thing like good enough. I think a lot of people do the two month version, it won't be that good and it'll kind of like slowly fade because it's like that doesn't really work. People pressure but, but I, the thing that I'm hopeful of is in doing that people also experiment with a lot of new stuff that, that it's not just like kind of to your point of like the beer app, somebody made the beer app. I guess the Beer app's kind of creative but like okay kind of a thing. But the first apps, they were all kind of the same stuff. It was all like some social media thing or whatever. But then people like there's all these other interesting things you can start to do with it, ways you can start to create variants of that where it's like some of them actually become pretty powerful and really useful. And so I think my hope is that people will experiment basically and like as. As folks now will have to go through sort of an experimental phase. There's a bunch of weird stuff that people are creating. Some of it'll be like, this is all bizarre and awful but. But people will uncover ways that like really help. I think Copilot's the first version of that where it's not a building an app that does this. It like kind of is a really fancy autocomplete where you can pseudo turn code into code. I think it's a good idea. Again, is there other things where you can like, it can start to. You can start to ask it more like how do I reason about this problem in a way that feels very natural. You're not having to go over to chat GPT and describe four pages of context, like here's the business problem. It's just like that sort of stuff where you have this sort of assistant that's going to guide you through how to think analytically. Somebody might with that. I think that'd be cool. Like that sort of stuff I think is what we'll start to see more of is introductions of kind of new points of injection into or like new people like inject this technology into various points in like how they work. Some of it will be all obnoxious and not very good, but some of it will be like, oh wow, this actually really makes things better. And then we'll all sort of like, you know, start to copy the stuff that works.

Okay, exciting stuff. Now Ben, as we wrap up today's episode, do you have any final notes or message to share with the audience before we wrap up today's conversation?

No, I appreciate, appreciate you having me on the main note is like, I don't know, I'm just like shooting a bunch of things from the hip here. So don't take any of this as me having any idea what I'm talking about. I assume that the future will look entirely different than anything we talked about today.

Uh, definitely. Everyone, I'll do the plug instead of you here. Check out Ben's substack releases every Friday. Correct. Ben yeah. Uh, yeah, it's really, really illuminating. Uh, it's definitely on my weekly reads list for me as well. Uh, so, yeah, everyone do check out Ben's substack. We're gonna leave it in the show notes as well.

Awesome. I appreciate it. And again, thanks for having me.

Thanks a lot, Ben, for coming on the pod. It.