AI Data Agents with Joe Reis - Episode #53

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast.

I'm Shane Gibson, and I'm Joe Reese.

Hey Joe, good to have you on the show again. Today I wanted to cover off this whole area of data AI agents or GenAI or LLM. And the reason I've got you on is because A, you're highly opinionated and call bullshit out rather than do the hype cycle stuff. You're independent in that you don't actually get paid by the vendors or us or anybody else like some of the analyst companies do. And more importantly, you run two podcasts, you're constantly at conferences presenting and talking to people, you do a shit ton of reading. If I had anybody that would be able to give me an overview or a broad view of what's actually going on in the market, When we rip in, what I'm going to use is a bit of a framing. I want to talk about GenAI, and for me, I particularly want to talk about LLMs. So my definition for GenAI for this session is large language models and how they could be used for data and how they are being used and what's bullshit and what's not. Let's start with a broad question. What are you seeing? What are the themes in the market you're seeing around this idea of GenAI, LLMs, and the data domain?

If you didn't know any better, you would think that every company and vendor and open source project is an AI company or project these days. I was just at Snowflake Summit last week and I came away with the observation that it was interesting. There's two— everyone's basically slapping AI onto everything. I think there's kind of two camps. There's the— I made a post about this too, where I feel like there's companies out there that really believe this is the next game-changing technology and And this is every business is going to have to adapt or die. And then the other end of the spectrum I see, it's a pretty big group of companies that I saw, not just at Snowflake, but others, other places. Yeah, this is, you have to do this because if you don't, it's, there's a lot of FOMO and it's not good optics if you're seen on the sidelines of this. There's the other camp that's doing generative AI because it just checks the box, you know what I mean? But then you talk to some of these people, I'm not saying this represents all of them, but some of the viewpoints I heard. Over and over was, this is, it's just a toy, but we have to do it. We have to play the game. So I think that those, that's definitely the observations I saw at Snowflake Summit. I expect Databricks to be very similar, though they tend to, you know, be a lot more kind of AI heavy, but for like actual AI use cases. We can dive into what I, how to distinguish those, but that's what I'm seeing, not just at Snowflake Summit last week, but I would say in conversations and travels around the world. Everybody's keen on quote AI for whatever reason. I think it was in Saudi Arabia back in March. Was it March? Yeah, it might have been March. I can't remember. I travel a lot. Anyway, it was, what was that conference? Leap. So it's outside of Riyadh. It's like 215,000 attendees from all over the world. It was a big conference, big. And AI was the big topic there too, right? And the Saudi government throws $40 billion into an AI fund to fund new initiatives. And you see this in places like Dubai as well, where I was back in the fall. Last year where they have a Ministry of AI and they have a Minister of AI. And if you ask around the people who are maybe at the ministry and might be in charge of having to try and do this stuff, there's, I think there's a lot of confusion still on what's hype and what's not. And like, what are the use cases for this stuff? And so I think to answer another part of your question, where are we? What am I seeing? I think that on the consumer end of things, the users, what I'm noticing is there's Last year was definitely, I think the theme was, let's imagine what we could do with this. Let's figure out what does the world look like if we were to implement this much in the same way the internet actually back when that the web was, I think, becoming more popular. That was a lot of the same kind of conversations. On one end, you had a lot of utopian visions. On the other hand, it was everyone trying to commercialize this as quickly as possible. And nowadays, the internet is so ubiquitous. We're talking over it. You're in New Zealand, I'm in the US, and that wasn't possible back in the '90s at least. But the conversations are very similar to, for users right now and consumers. If we're going to implement this, what are we going to do? So like last year was definitely the age of, you know, the year of imagination. I think this year is a year of POCs and trying to test it out and seeing where it actually has a use case in companies. And then I think next year is going to be the year where the rubber hits the road. Either you start seeing companies start getting some traction with this stuff, or I guess we'll see where it goes from there. But I think you have an idea, you've been around the block a bit. When the hype cycle fizzles out, then either there's either been proven out use cases or it's, I don't know, the hype cycle implodes and people move on to other stuff.

I'm intrigued by that. We've seen hype cycles. I'm old enough to see them around the OLAP cubes and then self-service BI and Tableau, and then the big data doopy wave. And then data mesh was a bit of a hypey cycle for a while. And now data products and data contracts, which I actually think will become sticky because I think they have value. And then with the downturn, we all thought we were going back to consolidation and end-to-end products rather than best of breed features, but then the AI waves hit, and now we see all the VC money going in, stealth after stealth companies starting to come out in the data space. And a lot of the time they seem to be just putting lipstick over ChatGPT, LLM, and creating these agents with no moats, but I'll come back to that. So I'm intrigued about whether this is just a hype cycle in data and analytics, or as you said, there's been some changes in the world, things like the internet. 'cause I remember the day when we had dial-up. I remember the days before we had that. Hell, I'm old enough to remember when we had video players with remote controls that actually had a wire to the video player. We've seen the internet, we've seen the iPhone or smartphone, we've seen a whole lot of step changes that have changed everything we do. And the real question for me is the LLM one of those or is it a Hadoop?

I think there's a lot of utility to it when you get into use cases specific to data. And we're recording this, what, June 11th or something? But this is also the week of WWDC, which is Apple's conference, and Apple just announced a slew of Apple intelligence integrations. So your iPhone, assuming you have an iPhone 15 Pro or above or an M-series Mac, will— you'll be able to take advantage of that stuff. So I'm curious to see what that looks like. In these cases that he said there, I was like being able to generate images and summarize texts and reminders and all this kind of stuff and calendar stuff. So I don't know, maybe it's a good productivity assistant. I use personally, I use ChatGPT every day, that and Gemini Pro, just because As I said on the other podcast too, I use it more as a conversational agent just to poke holes in my ideas. I don't use it to write. I think that, I don't know, defeats the purpose. Plus it comes up with weird results anyway. So you don't know you want to copy and paste that kind of stuff. But so my workflows, I use it all the time for just ideating stuff. I think it's awesome for that. It's like having a really smart friend that sometimes lies to you.

I think that's one of the keys is that the LLMs are non-deterministic, which means that you can ask it the same question 5 times and you will get variability of answers. And in the data space, what we really want is we ask the same question 5 times and we get the same answer 5 times. How many customers do we have right now? And if we haven't refreshed the data, the answer should be 42, 42, 42, 42, not 42, 43, 926, 1,233,000. So there's this idea that actually the LLMs are completely the wrong technology for data. And that comes back to the use cases. One of the things we found is our partners get asked by their potential customers, what are we doing in the AI space? And this is not because the customers or the organizations actually have a need for it. It's now, like you said, just table stakes. They want to know that we're doing something in it. And so I had to think about what would I do in the LLM space. And to me, I had to come up with a set of patterns because that's how we think. So what I did was I came up with 3 patterns that I understand. I talk about Ask AI, Assisted AI and automated AI. So ask AI is that chatbot feature. Ask something, get an answer, ask something, get an answer, ask it something else, get another answer, and then go and do the work yourself. You just, it's helping you think, talk, being, being that friend, but you're making the decision, you're doing that action. Assisted AI for me is that copilot behavior, which is, it's watching what I'm doing and it's helping me do it better, but I'm still making the call. And then automated AI or autonomous is, I'm not even involved. The thing's just doing it for me in the background and it may tell me, but there's no human interaction. So let's go through those kinds of patterns. So the first one is Ask AI, which is what I'm seeing pretty much every data vendor and every still startup doing is the text-to-SQL, that natural language. Let me ask a question, write the SQL, give me the data answer. What are you seeing in that space?

Oh, that's, if you talk about table stakes, I feel like that's something that everybody's trying to do, or at a minimum has included some sort of functionality into their products. As far as I can tell, everyone's got a copilot, for example, everyone's got some sort of LLM search. So when we talk about that, it's, it is table stakes though. If the, if the donut chart became like this hot thing that everyone was interested in, 3D donut charts, I'm just going to throw that out there. But then every tool now has to support 3D donut charts, right? It's not special anymore. It's just. What flavor are you going to use? What color? Are you going to have the kind of just like flies onto your PowerPoint presentation, like back in the '90s and 2000s? What flavor are you going to use? I think that's where we are with the search. Snowflake was interesting last week because they unveiled— I can't remember the name of the product— Snowflake Analyst, I think, for lack of a better way of describing it. But they have their own LLM now. So you can actually ask questions of your data in Snowflake. So all of a sudden I thought about all the vendors out there, like ThoughtSpot. And others like that. And I was like, okay, I think that might be a bit of a concern because that's literally what they do is search-powered AI.

They bought Mode, right? So after, after years of saying no dashboards, dashboards are dead, they went and bought a dashboarding company. So that was maybe, maybe they saw that the, the moat for natural language search is disappearing.

Yeah, it's interesting one. And then there's a lot of other LLM-powered vendors of that, so like Summit 8, um, I didn't want to ask. I didn't want to be mean about it, but I could definitely tell that it was, it's an announcement that you don't want to hear when you're the vendor that you're partnered with and you're at their summit basically just knocks you off with their own version of your product.

But we saw that with OpenAI. We saw people starting to build these agent capabilities on top of OpenAI. And then all of a sudden they came out with the ability to build your own agents in their tool. And it just killed the market.

That's what happens when you build on top of a platform like that. It's, they have all the intelligence of what's going on. What do you think is going to happen to you? That's how it works when you're too dependent on the large vendor. But it is what it is. It's like you say, OpenAI did that. And that's been the case since, since we've had large monolithic kingdoms that we're attached to. Yeah. So I think that's an interesting one. So to tell AI what to do or ask AI what to do, that's an interesting one. The tell AI what to do is an interesting sort of I think angle that is going to have some interesting impacts on software development and application development. I know Andrew Ng is working on like a lot of agentic workflows right now. He actually just announced open, an open source version of something that he did over the weekend. I know Andrew spends all of his time on the weekend just nerding out on stuff. So this is my project he did on Sunday. That's so basically give it, give it one agent and some instructions. Write the code, another agent will proof it, check it out, test it, another agent will deploy it, and basically the whole feedback loop, right? He's working on, in this case it was, I think, like language translation and having things checked, but the whole point is like you could extend that to basically use cases where you had various workflows that needed to integrate. And I think the success rate on it's been, that he's been building out and others like that. I know that Microsoft has their own Dev Copilot. I think it was, I can't remember the name of it. There's a white paper on it about a month or two ago, but that was another example where it was like, okay, this is where it's all going. It's just. These are the workflows. We can talk more about the impacts of that, but I think that's an area where you're going to see a lot more and more activity.

One of the other use cases you've seen a lot come up is the idea of using the LLM to populate, you know, the descriptions of all your data. We all know that one of the biggest pains in the butts about implementing any catalog tool was A, buying it and turning it on and then integrating it with your source systems so you can actually get the data in there. But after that, it was some humans had to go and actually update all the context about all that data because it never existed. I looked at that use case and I was like, yeah, okay, it makes sense. But then I went one step further and said, but hold on, what we're gonna do is we're gonna get the machine to give us the descriptive metadata for all our data. And then we're gonna go and ask the machine a text-to-SQL question, and it's gonna use that descriptive metadata as the primary input to that language. So now what we've got is pretty much the machine making shit up, asking a question while the machine uses the stuff it made up. And then you go, oh, that's fine. What we'll do is you'll see the vendors make it as a suggestion. Here's the text, descriptive metadata, you have to approve it. But what we know as humans is when people have a drossy job like that, they just hit the approve button 9 times out of 10, because they don't see the impact of that decision. I can see using LLMs for actions. I was listening to a thing around Baby AGI and the idea that you could give it a list of tasks and then that goes into the vector database. And then whenever you ask a question, it goes and looks at that stuff you've preceded to say, oh, if you ask me that question, I need to go and do these things and actually do the work for you. But that reinforcement model where the LLM is making shit up about our data and then using it to answer our questions, I still haven't got my head around how dangerous that is.

Oh, it could be very dangerous if it's not, if there's no enforcement of rules or any, any sense of correctness, right? I know Juan Cicada, he's done quite a bit of research on the hallucination effect of LLMs on Texas SQL, for example, and it's horrendous. It's horrible. I think low double-digit accuracy just on its own. And I've seen other benchmarks. I think Seguin Mira did something like that as well, where he benchmarked accuracy. And it ain't great. If you just use like regular ChatGPT or OpenAI, yeah, it's going to suck out of the box. And so you definitely have to help give it some context. And so yeah, it's still early days and stuff. Like I, I wouldn't use it in my own business. And the question I always ask around the world is, would you stick an LLM on top of your corporate dataset today? Sight unseen. And the second part of that question is, would you bet your job that it's going to work awesomely? Nobody raises their hand.

I suppose the third question is, would you let it do your job for you? So it answers for you without you even seeing the answer. Would you trust it that much?

Yeah. And I don't get the impression people are at that point yet. You're starting to see some indications of things like knowledge graphs and ontology mappings and stuff like that. Taxonomies even. Other structures really. Semantic layer, I think, was another one people are pretty keen on that could offer some benefits. So yeah, it's still early days and all this stuff, but I'm definitely not going to say we just need to throw LLMs away and avoid them. I think that they're remarkably awesome technologies, but we're just trying to figure it out. I was reading the Anthropic paper where they were probing the inner workings of LLMs and trying to figure out how to nudge it according to the weights that it used that attaches to concepts. So in this case, Golden Gate Bridge or something it was somehow fond of. And so they kept nudging it and so forth. I think we're just starting to figure out how these things work. The fact that like the experts who create these models don't also know how they work.

It goes back to the neural nets back in the data mining days, right? You threw it into a neural net and it gave you an answer and it looked reasonable, but you had no idea. It was not explainable back then. We tested it.

We didn't care that much. I was just writing about that in my book, the early days of neural nets and stuff. I think it was the '40s actually when it was first introduced as a concept, but it was a while. Yeah, so it's interesting. And, but the problem that's always existed with neural nets in particular is that it's a black box. You don't know what's going on.

What we used to do a lot of was when we're doing the testing of the different models and lift charts and all that stuff, often we'd see the best result out of a neural, but we couldn't explain it. So we'd actually take the customer through decision tree and we'd say, look, the decision tree is telling us kind of the basis of the model. It's not the model, but at least it's explainable. So I think we'll see LLMs starting to give us an explainability path where it's not actually what it's doing, but it's giving us enough of an insight that we can nod and go, yeah, that's what I'd do. Yeah, maybe, we'll see. So let's go back to that. So one of the questions I've been struggling with this text-to-SQL is we trust analysts that don't know what they're doing half the time. I've seen analysts doing averages of averages. I've seen people take data that has absolutely no quality and treat it as the truth. So we've always had this trust in these data people who may or may not be doing the right work, may or may not be using data that has quality, but we believe their answers. So how is text-to-SQL any different? Is it because we don't trust the machine or because we know it hallucinates that actually we trust it less?

I think because there's always been a degree of fact-checking that you can do with a human analyst. You can, you can say, tell me your assumptions behind this, right? And I think, and there's, there's a certain level of domain expertise that's implied with, with an analyst that you. Probably don't get with text-to-speech. So a junior analyst, for example, might perform, I think, technically at the same level as ChatGPT. I mean, there's that, but which one would you be able to ask questions about, ask questions from, and get a reliable answer back? I guess also, how would you know that you're getting the right answer? See, that implies some level of domain expertise on your own end when you're asking these questions, because I guess you could flip the coin and do the Turing test and say, okay, is it an analyst or is it ChatGPT that gave me this answer? I don't know, it's a fun game to play. I just thought of that. But I think, so that's where we are though. I think that the belief is that these things, these LLMs and so forth, replace analysts and software engineers. And maybe they could actually. I don't, I'm not going to discount that. I think in the next 5 years you might actually see that to a large degree, or at least supplementing the work in a much different way than we think so. But as of today, I don't know. That's a good question. I guess the Turing test is one way to maybe help evaluate that question. But it's an interesting question too, because with data and analytics, You pointed out earlier that datasets are sometimes horrendous, and sometimes you don't know they're horrendous. I always say data has been a silent killer in that sense where you don't know if it's good or not sometimes, and nobody does. And so you just use it. And I guess then people just start saying it's directionally correct. Maybe it's not completely accurate, but it's believable.

When we started adopting Agile in the data world, we started to pick up the idea of acceptance tests. And so let's say we're going to go and calculate how many customers we have. And we wanna take a test-driven development approach, then actually what we do is we write the answer of how many customers we have and we write the code until it gives us that answer. But you go to a stakeholder and say, how many customers we've got? Because I need to write that as a test. And their answer typically is, but that's what I'm asking you to calculate. So you get this problem where you don't actually know what ground truth is. One thing I am seeing in the text-to-SQL space I find interesting is a lot of the vendors now are actually producing graphs as a reinforcement feedback mechanism for the answer. So how many customers have we got? We've got 1,000 and here's them by region, here's them by product type so that a subject matter expert can look at those graphs and go, they look wonky. One customer in Northern and everybody else in Southern, we know that's not right.

It's like a CAPTCHA test for your data. That's interesting.

Yeah. And I don't know whether we're doing that because they found it's a great pattern for actually getting the answer or in our toolkit, we've always had graphs and we love graphs. So why don't we just give them back graphs as the best way of reinforcing forcing it. I'm with you. I think there is lots of opportunity. I think the models being non-deterministic make them incredibly dangerous. We haven't figured out how to solve the problem and maybe we get an iPhone moment, maybe we don't for Ask AI, we'll find out. But you do it for anything to do with text. I'm like you, I use ChatGPT and Gemini every day because for textual work, it makes me faster. I can ask it questions. I can have a chat to it effectively as a peer. And I find I end up writing better content or writing a better answer to a question I have. We gotta say the adoption of it means it has value. Otherwise we would've given it up like we have so many technologies before it.

Well, yeah, it's not like Bitcoin or blockchain where the blockchain paper came out in what, 2008, I think it was. Sometime around then, but there's still not that many tangible use cases for it that you can think of. I think you're, there's a lot of grasping at it and a lot of grifting, but that's not the same as having it actually— there's been time to play out where there would have been something 16 years later, right?

Yep.

A lot of people made money on it, but that's not the same as it having utility. A lot of people made money on blockchain and Web3 because it was a— because it attracted a lot of grifters.

I know lots of large consulting companies that made money on Hadoop because they tried to make every small business look like Amazon or Google. Not a great thing, wasn't for people, for those organizations, but it was the hype cycle.

But the interesting thing with generative AI, the thing is that I like about it, and I started a new business, started a new publishing company I'm going to be announcing soon. But I think it's an exciting time to start a new business. And I'm not just saying for content, but for all sorts of use cases. I'm trying to figure out, okay, so like, how can I basically, how can I create workflows where I won't need to hire people? Like, how much AI can I include in my company where it's just an AI-powered company, right? There's always been the speculation that somebody is going to come up with a billion-dollar company and it's going to be one person running it. I don't know, I'm willing to at least take a stab at that. I think that's pretty cool. So it empowers because now instead of you trying to like emulate the big tech companies with your dupe stack, now I can possibly do the work of several people that I don't need to hire anymore because I have workflows and generative AI just saves me that much time and money. So I think that's the goal. That's why a lot of companies think it's hot on this, but that's also ignoring the utility you can get from traditional machine learning as well, which also has a ton of utility as well. Having a background in machine learning, I think is So it's good for me now. I can think about, okay, now that I have a new business, how would this work?

Yeah, I think that background of science, that background, and you have a hypothesis and you're going to test it. And if it proves value, you keep it. And if you don't, you're going to throw it away and you're just going to go into constant experimentation quickly. You know, that true idea of agility. And then all you then end up doing is paying for tokens over people and they start becoming expensive as we're learning. Let's move on to assisted AI, which is effectively copilots. So we've got Copilots for SQL development and for software development. Starting to look at it differently in terms of for particular data tasks. So the example I use, 'cause this is what we've done in our product, from my point of view, we played with Text to SQL. I looked at it and I went, nah, we, we're not gonna go and push that anywhere near a production capability. I just don't trust it. I don't think it has the value yet. I looked at the augmentation of descriptive metadata and I could see how it saves time. But again, I was really worrying about that reinforcement and people being lazy. So I'm like, I'm not touching that bloody thing again for now. And what we ended up with is data quality rules. So the problem I wanted to solve was like all data platforms, we've got the default data quality rules, not null as a phone number. But as you'll know, everybody's data is of variable quality and you end up having to create these custom tests. And the process that I wanted to optimize, because I can't code, is the workflow that I had. The workflow I had was I look at the data, I go, yeah, there's a data quality problem. I need to write a test to tell me that, that runs all the time. That's what we call a custom trust rule. 9 times out of 10, it's some form of string-based thing I needed to test for rather than a number. So then I'd go into ChatGPT, say, I've got data in this format, use dummy data, tell me how to test it. It'll come back with a regex statement. It hallucinates. So then I'd go into a regex testing tool on the web and I'd put in dummy data and the regex code and make sure it executes. And when that was all happy, then I'd go back and actually write that as a test. And I looked at that and said, lots of steps, lots of handoffs. How do we do that? So we ended up using, we're Google, so it's Gemini, but using the LLM. So it looks at the data, it tells me what the data looks like. It says, I think this is a test you should write. Here's the code for the test. Here's what this test's doing. Here's it executing to make sure it runs and giving you the answer. Tell me what you want to change. And that saved me a shit ton of time. So for me, assisted AI are those use cases where a human's doing some work and the machine is now making that work faster or giving them hints, but they're still finally going, yes, that's acceptable. So the human's still in the loop. So what have you seen? What other use cases have you seen around that idea of assisted AI that you're seeing in the market or hearing it on the podcast or hearing from those conferences?

Yeah, it seems like every company now has a copilot integrated, right? Even Apple just announced that they're doing a copilot for Xcode just yesterday. I think they had dropped that. So like everyone's going to have a copilot of some sort. I think there's some vendors I've seen that have sort of AI enabled Copilots for making data pipelines. I think is interesting, right? Like, I want to create this view from these tables in Snowflake, right? So this might have some JSON here, you have to flatten it out in Snowflake. This might have some other stuff joined together and so forth. And so that's a data pipeline. I thought that was interesting. I think the interesting thing with Copilots is I'm going to— if I'm going to prompt for code now, if I do like a manual review or a pull request with somebody, Do I have to include the prompt they use?

That's the context.

So if now you're doing prompt reviews with people and it's okay, is this like the best prompt you could have used? If you're pushing code to production as a team still, which I think a lot of people still are, like Copilot will generate the code, but that doesn't mean I'm going to let this thing run willy-nilly. I still want to have somebody else review it, which is still common on engineering teams. And so I think that's interesting, but the, but these workflows didn't have the capability of the workflows and these tools didn't have the capability for prompt reviews. I was like, so. How would you know that what you would have written, you know, can be repeated? Because the last thing you want in data pipelines, for example, you talked about determinism, 42, 42, not a bunch of random numbers. You don't want your data pipelines to be the equivalent of a random number generator, where it just makes up a different workflow every single time. Like, that would not be ideal.

Okay. So, what you're saying though, is it's a config-driven pattern. So, you're pushing the prompt to production, and it's generating the SQL every time versus using the prompt to create the SQL and then pushing the SQL as the thing that's committed in each tool.

Either, either or. Yeah, either or. Cause some of these tools have both capabilities. I can make a prompt to create a pipeline or I can get the code to create a pipeline and so forth. Either one. So in either case though, now it's okay. So what was the quality of the prompt that you used? That was the first one of the things that came to my mind and my recommendation, one vendor was like, you have to have a way of at least tracking the prompts that are being used for this workflow. Right? If I'm asking it to generate code for me, what exactly did I ask it?

Auditability and traceability should be core of what we do. So every time you write a prompt, it should be recorded. But I would take that scenario, if I'm pushing the prompt to production and that prompt is generating the SQL on demand every time it runs, I'd call that automated AI because there's no human involved in the generation of that code and that non-deterministic capability is going to kill you, maybe, but you've got to solve it.

This is also where I think where the agentic workflows are going to be coming in handy too, is they'll just do the checking for you, right? But there's a high level of trust you have to have in that whole system to work. But I'm pretty convinced that's where everything's going is with software dev and everything else. There's a really interesting article that I think, I can't remember who it was from, some venture firm. I think it was actually Venture. I can't remember. Anyway, it was, it was really good because they had a good description of basically We have, I think the consequence of this will be very fascinating. You know, they talk about basically how software development is one of these things where it's so expensive, where there's actually a ton of technical debt in society as a result of the amount of software that's not written because it's labor intensive and so forth. But now with generative AI, it opens up the, I guess the cost of developing software could theoretically drop to almost zero.

Oh, look, the first person that generates your UI front end off Figma designs is going to kill it. Yeah, if you want to be a billion-dollar company of one, write that puppy. Because the amount of waste in terms of the front-end design we do in Figma to make things perfect, and then having to pay somebody else to then generate the actual UI code to execute that thing repeatedly, like that, that is a form of waste in my head. And it changes. So yeah. The other thing about that generated code is there are a bunch of people out there highly opinionated on this. And just to call out, thank you for last Friday's 5-minute post about don't be a dick. Yeah, it's good that when these things happen and they happen badly that people call it out. And the reason I mentioned that is I did a post on LinkedIn around this area and somebody came back and they're one of those opinionated people. And they didn't cross the line, the post that you talked about, but they got close. And their argument was the LLM could never write the code that was as optimized as what I can write it for. Like, here's an example and they haven't done the indexing or that kind of stuff. And I'm like, I don't care right now. I, I've got firepower in the backend and I can afford to pay for that firepower to run non-optimized code as long as it's accurate. So we gotta make sure we are focusing on the thing the human can add. Now, again, there's times you have to optimize that code for a whole bunch of reasons, but that's when you use a different pattern. And so talking about that, the, the problem I have with Copilots and data right now is they're so broad in that they're just writing code. Whereas if we think about it, they should be solving micro problems. So that example you said of create some views in Snowflake, we should be far more targeted. It should be create some dimensional views or create one big table views or create an activity schema view or create a view for Power BI in DirectQuery mode where I want the response to be under 5 seconds for a million rows and 200 columns because that's the things it can do is actually when we give it that framework, those set of patterns and that constraints, it's really good at staying within that boundary to a degree until it hallucinates. So what's your thought about that, going from these broad brush Copilots to really micro use cases?

I do agree with that a lot, actually. And I think it's a good, it's a good thing to bring up. I think it would help, it makes you, it forces you to write better prompts too, because it's a lot more specific. But I think, yeah, I do agree that there's definitely optimizations you can get that would be really hard to do because you don't have the context of all the systems. Maybe you don't, maybe you're like this guy that's just so amazing at writing code that he doesn't need anything. He could write it in Vim off the top of his head and that's, that's super cool. But yeah, I'm curious to see where that goes. I don't know. I think it's a good idea. It could open up a lot of, if you think about the workloads on a data team now, they're often asked to solve things with a very blunt instrument because they're short on bandwidth and time. And so at best, most data teams, they start out with good intentions and they start out solving great problems. But from what I've seen, the backlog increases to the point where you don't dig out and then you're trying to go for the easiest wins that you can get to or the loudest voices. Yeah, what I would be interested in is if you could go through and actually automate the clearing out of a lot of tickets because those tend to be very specific. You don't write broad tickets. That's the nature— well, maybe people do, but I think it's really bad form. Tickets, issues tend to be very granular and very specific. So I think that that's, to your point, an area where you could actually have a lot of real value.

That's a really interesting use case, especially in mixed models. One of the things we're helping a customer with was they are moving from a legacy data warehouse platform to a new modern data stack. And you know how that goes. It's been around for a while. They've got 1,200 reports, a whole lot of data, and undocumented to a degree, and they've got to figure out what they can decommission, what's got value. And they had a bunch of people working on it for 6 months to try and boil the ocean and figure out what the most important stuff is. What they determined was a lot of the reports in the legacy tool was the old copy the report. So Report A, number of customers. Report B, number of customers with a filter by region. Report C, number of customers in the last 12 months. So a lot of duplication. And what we did was we ran the report definitions through stuff we were working on to show them what was similar and what was different. But at the same time, we used some of our internal stuff and we actually ran images of our reports through to the LLM. As images and got some feedback. It was amazing what it did. It blew me away. It whacked our costs up massively. It was, we learned very quickly that multimodal image stuff costs you more than text. But there's an interesting idea there. If a user will often upload a ticket with a screenshot of the problem, this report, that number with a red circle. So again, there's a really interesting assisted AI use case there, isn't there? Where it just scans it and says, looking at that image, this is what looks like the problem. Looking at the code base, this is where we think The problem is data person go in and see if we've got it right and fix it.

I think that's awesome. Yeah. And I'm really excited about stuff like ChatGPT 4.0, for example. I think that at least demoed really well. I still need to try it. A lot of the features haven't been released yet. So I was thinking about just how do I use it to help my kids with their homework and that kind of stuff. They had this awesome demo of Sal Khan where he's with this kid and it was basically walking through how to solve a, I can't remember if it was a trigonometry problem or something like that, but that was pretty cool, right? It just looks at what you're doing and says, oh, here's some suggestions on how you could. Fix it. And so that's, I think to your point with multimodal, that's also the stuff I'm excited about is like what opportunities that actually open up in the business. Cause right now we've only been talking about text, but there's a lot of other stuff, right? Explaining things, audio or video, maybe looking at your reports and saying, here's what I would do. And just given the context of your business that I've learned about, I think there's a lot of really cool stuff that could, I think it will be happening. I don't think it's a question of if.

We can steal from our software brethren. A lot of the tools that record users' screens in the software application, as you're building it, you plug some stuff in so you can see what they're using in your product and where they're struggling. A lot of those are now using, and I'll use AI because I'm not sure what they're using under the covers, but yeah, it'll actually go through and say, this screen, this percentage of the users seem to be struggling in that area. How often do we actually do usability testing on our reports? Never. It's funny that our reports are effectively small apps and we actually do none of the software engineering work in that. Very rarely are our reports code. We're always dragging and dropping or something. We never version them. We do a little bit now. We don't do any of the test recording and logging of who's used what. And we certainly don't use some of that software engineering stuff. So it'll be interesting bringing in those other techniques that we haven't used in data before.

Oh yeah, I think that's what I'm saying. I think it opens up a whole new world. For the longest time, especially just reading the original business intelligence paper, it came out in 1958, first time the term was used. It's not a new concept. That's a retiree on pension right now.

Semantic layers, right? We've seen those before a few times. They had value and we keep losing them. Yeah.

And so it's just all this stuff. And I'm okay. So we've had this workflow that's existed for a long time, this paradigm of looking at the world and how to view it. And it's been successful for the most part, but what else is there out there that we haven't thought about? What, how is there another way that we could be looking at our business? Apart from the traditional ways that you either successful with or you mostly struggle with, to be more to the point, with a lot of companies. Like data warehousing and BI, for example. Why is it in this day and age that we're still struggling with this problem? It's been around for ages, like literally a long time. Bill came up with it a while ago and moved on from that actually a long time ago too. You're like, this is pretty boring. I'm going to focus on text, not structured data. But why are we still struggling with these problems that have been around for decades? And yet, We're also at this inflection point where we're introducing these extremely complicated, almost alien workflows into our businesses. That's a dichotomy I'm still just trying to wrap my head around really is you have all the problems of the past, namely in analytics and so forth. And I've been through a lot of efforts of trying to shoehorn machine learning into companies. For the most part, it doesn't work. They're not ready for it for a lot of reasons. But now we're at this point where it's okay, but then you have a generative AI, which I said it's alien. I literally mean that. It feels like you're talking like a space alien sort of came to the Earth and now you're trying to learn how to talk to it and it knows everything and sometimes it lies to you and so forth. But it's an interesting one. But now you're trying to shoehorn that into businesses too. But the dichotomy is obviously that a lot of the classic data problems, we haven't even figured those out. So can this help move that along? I think that's maybe the only option you have. The other approaches haven't worked.

It's an interesting question. Again, I've been doing this for 30 years and I'm with you. For 30 years, we've used different technologies to try and solve the same problem and we haven't. We still have disconnected data teams. Stakeholder asks for a number, it still takes us days, weeks, months, depending on if we have access to it, to give them the number back. We can't prove the number's right 9 times out of 10. Will this change it? I don't know. I go back to the iPhone moment, you know, because there were smartphones before that didn't change the way we behave. Maybe it is the technology will generate this new thing that we've never thought about. Uh, I just don't know what it is, right? So I'm still focused on, okay, I'm not going to do that big thinking. I'm just going to focus on small problems where it can take things that used to take me 3 hours, or when I had to go and learn about it, and let me do them in a minute or 10 minutes, or teach me something I didn't know.

That's exactly it. I mean, you got to, if you look at what machine learning has always been good at, right? It's taking problems that humans can't scale to in solving those types of problems. So if you and me tried to look at a billion pictures and label the cats in the pictures, like, we would die before that happened. That's just jotting it down by hand. That's cat, no cat, hot dog, no hot dog. We would literally run out of time. Machine learning algorithm has been able to do that for ages, and it can do that at scale. Million, billion images, not a problem. We'll do it. Where problems like that are good. So with companies, for example, right, you talk about the team that's migrating the data. This is classic example of something where machine learning could be a good fit, where it's at a scale where it's going to take humans way too much time to do. It's not really value-add at the end of the day. It isn't. And then the other part of it is you have technology now that with generative AI, generative AI is really good at either synthesizing and creating data, new datasets based on input, or summarizing. These are the two massive skill sets that it has. This is perfect. Right. So I wrote an article back in November about this where I felt like AI is probably going to be the— this is probably how a lot of the data management problems that have plagued the business world for ages get solved because it's going to operate at such a scale that it would have to work, I would say. Because if you do— if this doesn't work, I'm not sure what will, right? If you're not throwing machine learning or AI at the problem, I'm like, what are you going to throw more people at it? Or are you going to throw more data governance rules at a company and tell them to go do that? Is that going to solve the problem? Are we going to put another post on LinkedIn complaining how the business doesn't get data and we just need to fight harder to get the business to accept us? I think that those are all stupid things to be talking about at this point. Like we have the technology that can be tweaked and honed to start solving these, these data management problems at scale.

Oh, and I think while we're talking about assisted AI before we get on to automated, yeah, there is some dystopian views out there and they are highly possible. You obviously know about Amazon Turk. Yeah, you can go and put all those images there and people in third world countries that get paid bugger all, go and do all this manual work to go and do what a machine can do now. And effectively that market's going to disappear, whether you agree it's a good thing or a bad thing, it's gone. That income for those people is going to get taken off the table. That could happen to the data people. It could. If I'm a stakeholder and I go, hold on, how's this work? I ask a question, even with these tools, it still takes a couple of weeks before my Jira ticket gets picked up and then somebody looks at it and then yeah, there's an analyst involved, but actually I don't trust their numbers any more than I trust the machine. Frig it, I'm just going to go and do it myself in 10 seconds and the number will be good enough because I'm making a decision and a quick decision on reasonable data is better than a long decision on 100% data, 9 times out of 10, right? There's certain use cases where that's not true. So that speed to market, I think is potentially the thing that will kill data teams.

Yeah, it could. It'll either replace them or hopefully evolve them to becoming just do more of what they should be good at, which is being domain experts in the business that happen to be also good with data. I think that's like, that's the role that I had when I got my first job was actually doing that. I had good data skills, but I was working in the business. I wasn't— now that we called ourselves data, I wouldn't use that word that often because it wasn't really relevant to the people I was working with. They're like, yeah, that's cool, you do data, nobody cares. But what I would like to do is increase my sales. What I would like to do is effectively allocate my marketing budget of millions of dollars a week to the most impact. Can you help me with that? I don't really care how you do it. So I think that's a role that people are going to be more evolving into in data. But I think you're absolutely right. There might be infrastructure engineers behind the scenes at companies that require it to maintain the infrastructure, but a lot of that's already been SaaS-ified away a decade or more ago anyway. Snowflake, you don't manage the infrastructure with that. They do. And yeah, it'll be an interesting one. The notion of a data team though, I think I've always had trouble with it. I feel on one hand, they're necessary. On the other hand, it's depending on the type of company you're at, and it's a broad generalization for me to say this, but what do most of them do?

They deal with the fact that if we're building our own software products, we don't care about the data. So we leave half the job undone. So somebody else has to pick it up. And if we're buying 12 different SaaS platforms, we don't worry about how we're going to integrate the data between them to answer questions on the combined data. Data teams end up picking up the technical debt for those two decisions. Which is we are gonna go put stuff in quickly and ignore the, the, the next bit that we need to do. So if we get back to, to the last one, automated AI, and, and this is the one that I'm intrigued with because originally what I'd say is it's, it's definition is machine does it and we're not involved. And the example I used to use was anomaly detection. And I use this one particularly because data comes into the platform, the machine looks at it, determines that the data coming is out of a certain range. So these are an anom— And when we first started building that, we were clever and we used K-means and some clustering, and that was all good. That cost us a bit of money to run and we were hyper-focused on that. In the end, we moved back to a bunch of group buys because actually it gave us pretty much the same answer and it was a shit ton cheaper to run. Now we could push that through an LLM and maybe it'll give us a better fit. I don't know. But the question is, it's still not automated because it's still assisted, right? It's telling me that something's wrong and then I've got to go and fix it. Automated AI would be, it saw an anomaly, it actually went out and fixed the data, reloaded it without me ever doing anything and told me it fixed the problem with that load. And the whole reverse ETL wave and all that hyper bullshit actually will come true with LLMs because what we want for automated AI is we want it to actually take the action for us. We don't want to tell us the customer's churning. So that we can go and figure out a save campaign so we can then type something into an email, automated email system to send them the offer. We want it to go and say, that person looks like they're going to leave. We know that the next task list is send them a save campaign, email sent, we're tracking their response. Which if you think about it was what we tried to do in data mining and machine learning 5, 10 years ago was automating that workflow, giving them that instant recommendation on an e-commerce site. So I think we're going to come back into that. And I think that's where automated AI will take us.

I love this version of the future, by the way. This is the world I want to live in where things are just done on my behalf. And then maybe you tell me what you did if I need to know about it. My first experience with this where it really hit home was I was working at a, we were an IoT company and companies kept asking us to do for real-time reports on stuff. And I was like, tell me what you're going to do with this report that I give you. What's the action you're going to take? So, I described the action and I'm like, "How about if I just take the action for you when that happens and I just give you a report on the things I just did?" That would be awesome. That's going to be a ton of time. And so, that's when I had the realization, this is my 2015 or something like that. I had the realization like, "Okay, this is how the world needs to work, right?" Where it's, if you can automate something heuristically or with AI, doesn't really matter, just automate the action that you're going to take. Especially as we move more towards what I called the live data stack in my book, right? Where it's convergence of applications, data, ML, all in real time in a real-time feedback loop. I mean, this is happening as we speak. I think this is the next shoe to drop. But as you get more towards that world, what you just described is necessary, right? Automated ML or automated heuristic workflows. That's how it's going to have to work because things are happening at such a speed that humans can't respond, right? But that's always how it's been. If you look at the nature of things like charts, for example, and analytics, analytical data is the only data that's actually meant for human consumption. I.e., it's something that you look at and you try and understand and make a decision off of. But most operational analytics, for example, isn't like that. What are you going to do if you saw a chart that updated every nanosecond? How would you possibly be able to respond to anything that happens in that time frame? A machine does that. And so that's always— but that's always how it's been since even since the days of mainframes. You did that because you're trying to do things at scale that humans couldn't do faster than humans could do it. That's something that they It's been around for '50s and '60s. So the desire of automating stuff has always been there. Now you just have machine learning where you can do things that can't be done heuristically. So I love this world, Shane. That's the world I would love to live in.

Yeah. I mean, the operational analytics one I used to love was contact centers when they were humans and you had the big TV on the wall with the call times, response times, the answer times, the people in the queue. And that was all great to know that you're in the shit, but what do you do about it?

Like, exactly.

You're gonna, you're gonna go and get 10 more people just to turn up off shifts into the call center now to take that spike because you are an important customer, but we have an unusual amount of volume right now. You're probably going to do some predictive stuff to guess what that spike is, but you may get it right or wrong. What are you going to do? You're going to cut people off and pretend you didn't answer it. You're going to tell the agents to answer the question quickly. You know, what action are you actually going to take when that graph that your queue, your queue volumes have gone through the roof? And that potentially again is with things like LLMs, can you ask the customer to tell you what their problem is and then push them to another channel? That multi-channel has always been the nirvana for this space.

Or just replace the agent with, yeah, or the customer service agents with an LLM or whatever and just be done with it, which is what a lot of companies are trying to do. Or it's either triaging or outright replacing the workflow. But yeah, that's where it's all going. I have no, no disagreements with that. That's, but it's the interesting thing is going to see what, what if this stuff does take off when companies are able to integrate this at at scale, like what happens to all the employees? I guess it's the central economic question around the world, but we're going to find out real soon.

Yep.

So that's— so I think it's exciting in some ways to be able to watch all of this unfold in quote real time, no pun intended, because that's not too often you get to live through an inflection point like this. And it's going to happen fast. I think if it happens, it's going to happen real fast, but then it's going to happen slow too, in the sense where it's like for the companies that have adopted it and it's a good fit, things are going to happen quick. For the companies that where their data is an absolute horror show, unless we find a way to use like an AI agent to automate the fixing of that data. Yeah. I don't know what happens to those companies.

My suggestion is there will be an iPhone moment with this. I think we've seen with the LLMs, the amount of usage that we do and everybody does, there is value there. We just don't know what the final value, what that unique thing that will come out. And so with that level of uncertainty, if you're an organization trying to make an investment, you're in a hell of a world right now because Everything's experimental. You've got to actually look to use it. Otherwise you will be left behind. Although I remember when I was at Oracle, it was, you're an e-commerce business or you're out of business. And there's still a shit ton of businesses that aren't e-commerce businesses that have been around for that long. But I think this will give you competitive advantage at some stage. So the question is, how do you make small bets? Yeah. And I go back to, if there's something that takes a team an hour, and, and the, the AI data agent makes it 10 minutes, uh, and doesn't increase your risk, why wouldn't you invest in that? That just makes sense. But just be realized that if it's a VC-funded stealth company, potentially it won't be around in 6 months' time for that use case, but there'll be a bunch, like you said, the platforms will take over those use cases because why wouldn't they? And then if you're a data person, it's almost go back to being a BA. Go back to that. You talked about that breadth of understanding business, understanding data, and now introduce the idea of understanding language. Become— figure out how to use those prompts as natural language to help you bridge those two domains. And then it's just another tool in your toolkit. It's another thing that adds value, and use it if it does and don't if it doesn't.

That's just it too. It is just another tool in the toolkit. It's not a godsend, and in some cases it's better to if-else statements will do just fine in your code, right? Classical machine learning will do just fine. In other cases, generative AI will do, right? So yeah, just, it is another tool. I'm glad you say that because I think a lot of people conflate generative AI with all of machine learning and AI, and these are not the same things, like literally not the same things. But I think there's, as I posted today too, there's, for companies out there, I would say just especially around the consumer end of buying these tools, just beware. There's a lot of charlatans out there, a lot of snake oil. Salesmen who are trying to pitch you on a lot of things that aren't true, right? This is a very complex topic. It's a complicated field and there's a lot of people out there who want to grift and take your money. So just make sure you understand who you're talking to, right? Not everyone's an expert in this stuff. I would say not a lot of people are. If you imagine, okay, this stuff's been around, ChatGPT exploded on the scene about a year and a half ago. Okay, then suddenly within a year everyone's an expert in this field. Are you kidding me? Most people I know that claim to be experts, they read like Wikipedia on something or barely even did that and like read the first paragraph and now they like can barely spell RAG or AI. So just understand who you're talking to. Go with people who have experience in this field, who have done the work. I think before and after the ChatGPT, because there's a lot of value to having been around the data space before all the generative AI hype. You just know how gnarly the problems are in a lot of companies. And they're, it's a junk show at most companies to put it very politely.

So I'll reinforce that in terms of when you get those demos, and the demos look beautiful with their curated data and their curated prompts, then the theory is this is nirvana. You can just throw your data at it into the magic sorting hat and it'll give you an answer. Make them do it. Because I guarantee you ask me to do that, I'm going to go, I'll do it, but I don't trust anything I get back. So let's be upfront about that because I know it's non-deterministic. I know it just doesn't work yet.

And that should be the answer when you hear an answer like what Shane just gave you, where it's an honest answer. Right. It's not a bullshit answer. And that's the kind of person you want to go work with, right? You don't want to go work with the person who's like, yeah, this will totally work out of the box. Don't worry about it. Your data's an absolute mess. Fine.

It's all good. Magic happens here. If it does, that's good, right? Because then I can be that billion-dollar company and retire, which won't happen, or I can go find another career. I need to know now. So if you've, if somebody's got the magic sorting it and magic does happen here, just let me know so I can stop. Excellent. Good chat as always. Uh, I think we should probably get together in a year's time and, uh, do the postmortem on GenAI LLM and AI data agents and see where the world is.

I think that'd be great because it'll be around the time when the next Snowflake Summit and next Databricks is happening. So as I put on my post today, I'm very curious to see where things are these next two summits next year, this time next year. It's not a recap. I think that'd be awesome.

Key prediction from you then. Next year, will Databricks Summit and Snowflake Summit be on the same day? Or a different day?

I think it worked out pretty well this year. So hopefully they're just different weeks for the vendor's sake. I think that's super convenient. I did last year when they were on the same week and that sucked. So I go to Vegas and then to San Francisco, but no, I think it worked really well this year. And Lord knows San Francisco needs all the help it can get in terms of tourism right now.

Let's do it in a year. Uh, let's take the transcript from this one and in a year's time, put it through whatever LLM we've got at at the time saying, if we were going to run a podcast that tells us a year later what actually happened, what would you predict? And then we'll have a chat and we'll compare it and see, uh, whether what it thought we would talk about is what we actually talked about. That'd be intriguing.

That'd be awesome.

Excellent. All right, thanks for your time as always, and, uh, I hope everybody has a simply magical day.

All right, take care.

And that, data magicians, was another Agile Data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.io.