Building Data Services with AI with Jason Taylor - Episode #79

Welcome to the Agile Data Podcast, where we talk about the merging of agile products and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, "Hey, that's not what I wanted," and then you hear your team reply with, but that's what you asked for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the green book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

I'm Jason Taylor, or you can call me JT, either one's fine.

Hey JT, thanks for coming on the show. I think today we're going to have a bit of a chat around building data services with AI, but before we do that, why don't you give the audience a bit of background about yourself?

Yeah, sure. I guess the simplest way that I usually explain it is beginning of my career was more quant research and data, and then I just gradually went towards the data. I don't know if that's towards the light or towards the dark, but, um, let's see, I worked at FactSet, Palantir. Usually everybody wants to talk about buy-side, a whole bunch of different places. Now I play in startup land because I have a death wish. I don't know. Yeah, working on a bunch of fun stuff.

What made you come from quant to pure data?

That's a good question. And quant is a data gig in a lot of ways, right? And Joe restarted in this as well, doing more quant research at one point in his career. Like, I think There are different types of quants, but I think a good majority of them, and as people have become more technical over time, they've become more data-oriented. Almost like sometimes I joke that data scientists are just quants, or quants are just data scientists with more subject matter expertise. Like it's all kind of the same thing. But why? I think the very practical answer is I don't have a PhD and competing against PhDs sucks.

I think it's interesting. It's this idea of breadth versus depth for me. I see people start out with heavy statistics, right? That's what they love and that's what they get into. And that's a very specific set of skills. And then often they'll go into breadth. They'll extend their skillset out and become more data-y and a little bit less steady. And my perception is because that's where the market is. It's easier if you've got a broad set of skills to be able to pick up a gig or do a role and do more of the work end-to-end yourself. Than if you're a specialist with a really deep set of skills. I hadn't quite thought about the PhD side. So I suppose I always talk about, are you going to be in the top 5% of your skillset? And I suppose in that area to do that, you have to be a PhD.

So first of all, let me start by saying I very much subscribe to the thing you said where I want to capitalize on my strengths. And there are certain things I know I'm good at. And actually Google came out with this cool tool where you can actually talk to it about a job and it will tell you, it's all AI oriented, but it will actually tell you attributes and things. I think you like to self describe your attributes and it helps pinpoint potential jobs that it believes in for you. And it's actually really smart. And I very much subscribe to this of attribute-based or characteristic-based job placement, if you will, as opposed to like, you know, there's a ton of people that are like, oh, I want to be a data engineer or a data scientist. And that's great. Those roles are wildly different at different places. What things do you like to do? What attributes? So long-winded way of saying, I very much subscribe to this. I think that there are certain things that I gravitated towards in terms of the attributes I like and the activities I like. Like, I like being analytical, sometimes to a fault, which I think we all do in the data space to some degree. But then I also think the PhD thing may be a little bit of terrible history, but yeah, at some point in time, from a finance trading perspective, people started leaning into math. Go figure. And there's portions of finance that have always been in, in that space. But at one point, I think it was in like the '80s, if— yeah, more the '80s and maybe early '90s was when they started pulling in like physics professors and the like, because people started looking across the aisle effectively and saying, hey, you're doing signal processing and studying these things, and hey, hey, that's the same as finance, like it's the same thing. So people started looking at that type of math and that type of process that they were doing. And that's, I think people started leaning into that, right? There's a piece of this, which is, I feel, and I'm sure this will relate to AI really fast, but it's just stereotypes, right? Like people are like, oh, I need a PhD to do this. And I'm like, do you really? Do you know what you're doing? Do you think all PhDs are the same? But ultimately that's a good chunk of the market. I'm not going to discredit people having a PhD. I think that's awesome. I definitely feel like I would have wanted to do that at some point in my life had I not made certain decisions. But yeah, it's hard. Those people are smart and it's especially on paper from a recruiting perspective, you know how it is. Some recruiters are just like, oh, here are the qualifications I was given. Yay or nay.

I think, I think the recruitment process is broken. I think that whole industry is going to be disrupted. I think we're going to move to more community or closed network based recruitment, very much based on recommendation.

Aren't we already there?

Yes, we are for some, but yeah, if I look at over here, the way the government over here recruits, my standard joke is somebody in a government agency generates a job request, job description with AI, to which then the candidate candidate uses AI to generate their CV to match the job ad, to which then the recruitment agent then uses AI to see whether the CV matches the job ad. I was talking to somebody and they were advertising for an administrator, office kind of administrator as part of their process. And they were saying they had 5 perfect CVs, perfect matches, all the experience, all the skills. And when they went to interview them, none of them had done anything near what the CV said they had. So I think this personal recommendation, this personal network is going to become more and more important. And the idea of skills being transportable, I remember many years ago when I was working at SAS and I was watching one of the projects, one of the consulting teams did for a supermarket. And they ended up using survival analysis to figure out placement of takeaway hot meals with something else. And I, Didn't do well at stats at school. I didn't enjoy it. So it was always funny when I went to work for a company that was pure statistics and I was like, how does this work? And they said, imagine you got a tall man and a short woman and they have a child. There's statistical statistics on which genes are going to survive and will the child be tall or short. That's what we use to decide whether we should put beer or Coca-Cola next to those meals. And I'm like, I still don't understand it, but it makes sense. And so there's this idea that actually, like you said, with the finance stuff, these skills you get in another domain, and then they are really useful in an alternate domain. So focus on the skills, not the role, and that's where the value is.

Yeah. Yeah. No, 100%. And I hate that that's perpetuated. I hate that the role-based mentality for a lot of people is somehow perpetuated. That's just how people think about career.

I'm currently perspiring on my second book, which is around data persona templates. And so one of the things I'm doing is I need examples for the book. So what I've started doing is downloading job ads for data professionals. And then I've created basically a ChatGPT agent based on the book, a whole bunch of prompting. And I'm telling it to take those job ads and create the persona template for me. And the persona template is skill-based. It's all about skills. It's really intriguing to take all these different jobs that look different, and then when you boil them down to what is the persona? It typically comes up with only 3, 3 core ones. There's variations on it, but I'm intrigued by that. But this one's not about data personas. This one's about building data services with AI. Why don't you explain what the hell you mean by that? And we'll go from there.

What do I mean by that? There's a very easy rabbit hole here, which is like, what is a data product? Which I'm going to avoid. We are not going to go down that hole.

Aw, come on. Pedantic Semantics says 45 minutes of us arguing a definition of a definition.

Terrible. No, I'm not playing that game. What are definitions?

Shame on you.

Define definition. If you ever see Joe Reese, make sure you ask him to define definition. Joe loves debating semantics if you didn't know that already. So if you see him or hear him, please ask him these questions. He loves them. Anyways, no, I'll start with an interesting divide that I was actually talking to someone earlier about. I always find it interesting because the vast majority of the data space, usually when you talk to other people in data professions, they're usually in some sort of supporting role, I'm going to call it. And when I say supporting role, I mean that the data itself is not the revenue generating aspect of the company. Very basic examples. I work at a company that sells ice cream, so I help them figure out how to best sell ice cream, but still the main thing that you're selling there ice cream, right? There is an entire world of people like us that sell data, and I've lived in that world for a very long time. It largely focuses around things like finance because finance is an industry that has to consume data in order to operate. There are other industries, I think marketing and advertising, things like that, are also in, in this scope, but there's an entire industry of people that just sell data. So I think that especially from a data services with AI, I think of a couple different things. There are The people that are pure enrichment-based, send me your data, I will do something fancy and send it back to you. There are the people just selling the raw material. Here's the data I collected in some capacity. A lot of those people are doing web scraping, not all of them. And then there's other people that do integration style work. And if you think about Bloomberg, which is relatively a household name at this point, they do all of them, which is interesting, right? They're both consuming from the public domain. They also have specialty knowledge around how to integrate the data and then also how to enhance it if you give it to them. But now we're in a new world, not to say generative AI is brand new. I think it's relatively common knowledge, maybe not for everybody, that generative AI has been around for a little while. It's just now very mainstream, very accessible. There's 10 kajillion startups around it, but it's very interesting because of course the key aspect of it, which is it's non-deterministic. So now I'm not selling a predetermined thing to some degree. I'm selling something that may be variable at times. I think that especially services around AI today, there's definitely no shortage of web scraping companies. I think the one that's exploded the most by a massive amount has been unstructured to structured data parsing. That's exploded. And I feel, and I'm curious of your opinion, and I'll pause here. I feel like that's a solved problem now. Maybe that's extreme. Do you agree?

I think it's like when people tell me data collection's a solved problem. And every time we onboard a new customer, 2 or 3 of their source systems, the ones we know will, and then there is always an outlier system where there's 100 customers in the world and the APIs are really badly documented and the data structures are a nightmare for us to try and understand. And I sit there and go, solve problem my ass. I just want to go back though, there's an interesting one. One of the things I do when I kind of work with companies and I'm helping teams out, we start this idea of this playbook. And the idea of a playbook is basically a bunch of slides and they have two purposes. The primary purpose is to explain how the data teams work for any new data team member. So when you're on board, you read this and you get a feeling of how they're structured, how their workflows, what the culture of the team is. And the second one is if you're a stakeholder in the organization, it tells you how the team works. So you understand how to engage with them, how long you're going to pretty much wait, what your role is in that data work. And one of the slides that becomes really common for me now, right at the beginning, is saying that I can categorize use of data in 3 ways. Internal use to understand the health of the business and grow the business. Data to support the customers where the customer's actually accessing their own data. So typically it's software as a service or a bank or insurance company.

Sure.

And the third one is where data is used externally for monetization. And that might be selling data, that might be enabling partners to use whatever you have. If you're an insurance company and you've got insurance agents out there, then I treat that as external data. You're giving them access to that data outside your organization based on your customers to make more money. So I kind of like that framing. So what we're talking about when you talk about data services with AI, you're talking about that last one, data being exposed externally, securely to make money somehow. And it may not be selling data, but it's definitely, we're exposing that data and sharing it to monetize it. Is that what you're talking about?

I think that is definitely correct. I also think it's a middle one too, which is like exposing their own data. If I heard that correctly, the exposing their own data, I think is another one like very common is just enrichment. It's already my data. You may be adding or doing something to it that then I'm monetizing the process as opposed to the data itself, if that makes more sense. Yeah.

Okay. I heard a podcast a while ago that intrigued me and it was around a massive US company. That had digitized and augmented all the lawyer case study, noti case history thingies. I can't remember which one it was. I seem to think LexisNexis, but I've probably got that wrong. And what was interesting here, what was interesting was, and it's coming back to this idea of semi-structured or structured, what they were saying is, huh? You define definition. Actually, I have a definition of it. And then I was doing a group thing with Ramona and Chris Gamble and Ramona and I have a disagreement on the definition of structured and unstructured. All around CSV, right? Anyway, so what's interesting about this is you think, okay, we've got all this case history stuff and it's in books and there's probably digital versions of those books. And so it's a sole problem now. I could go and grab all that content and digitize it and create a service that competed with them. And yes, they've got market share. How do I find the market in that potentially? But I get the impression it's not cheap. The lawyers, nothing with lawyers is cheap, so I could probably disrupt them and Uber them or Netflix them, right?

Yeah.

But what the key was, the augmentation. It was the many years of highly skilled and trained legal professionals augmenting that raw data, even though it was unstructured, with context. And that context is where their moat was. So before we get onto that idea of, is the problem solved? Is that what you're saying though? Is that once you get this data and then you add additional context to it, that creates the value, that creates the moat that's harder for anybody else to actually go and breach?

Yeah, I'm just going to start outright by saying, yes, I agree with that. And I think that's, again, I'll refer to Bloomberg, FactSet, S&P, et cetera, because that's the world I know, which is public company filings are public. Right. You can go and you can download Apple or anybody else's 10-K or Q or whatever other filing. Cool, neat. There is some value there in being able to access it and make it easy to consume and blah, blah, blah. No discount to that. And there, there's actually cool open source people doing that exactly now, which I'll come back to. But yes, the work that they do to organize it, right, to your point on semantics and governance and all this stuff. It's that organization that actually gives value to the data. It's not about just serving it up. It's about making it usable with other stuff or potentially integrating it, so on and so forth. So, and is that all hardcore domain expertise? Uh, some of it, not all of it. I imagine in the legal space it is far more, but the part I want to come back to is that getting access to the data and the value provided in that just operationally. Dramatically lower, massively lower. I don't think anybody disagrees with that. I think that, that integration, that domain expertise has also become more accessible. I think that even though in that particular case, and maybe it was LexisNexis or something, has all these domain experts there, I do also think that LLMs have come a long way. They can infer a lot of these things. It doesn't mean they're perfect, but it does mean that We've accelerated what usually took hundreds or thousands of individual people to fill those gaps. So the moat has diminished significantly. Do I think that there's still differentiation or IP and domain expertise? I absolutely do. The things I've been talking to people about, like, I, I'm very comfortable with that idea. There's a very basic idea, right? Not everything we know is written down, period. It's just not written down, or it's a little bit more out there. And at the end of the day, large language models, for the most part, gravitate towards the mean. That's by definition what they do. Do I think that means they can't learn these things? Not really. And I'm using learn very loosely in that context. Do I think it means they can't learn these things? Not necessarily, but I do think the tribal knowledge, et cetera, this is what people are trying to do with RAG and everything else, right? It's just, how do I shove my tribal knowledge into the thing so that it has the things it needs to do this? But yeah, that is definitely what we're talking about. I think that the expectations and obviously what you can accomplish today is wildly different. And I think that especially from what you can do, the bar is tremendously lower. But I think the expectations are wildly off the chart. I think to the earlier point, unstructured versus structured, there was a period of time when shoving a PDF into pick your favorite model, whatever that was, sucked. A lot, even if the PDF was pretty modern, you could extract the text off. Great, cool. Still not great. And we've come a substantial distance from that where I, on a regular basis, am processing PDFs just as things I'm doing as part of my day-to-day. And I really like Gemini. I will plug them. I don't mind. I really like Gemini's Pro model for the vast majority of things that I plug in there. And bear in mind, these are all kinds of interesting financial statements from a variety of different providers. It could be org charts. I've thrown org charts in there, which has been cool. Diagrams, all kinds of weird shit. And they, with remarkable accuracy, just pull it off. And my favorite part is that, and not that I'm doing this professionally, but I think the first thing I always try and do just for absolute giggles is I drop the document and I say, give me everything. And then I'm just like, let's see what happens. Fuck it. Let's see how good or how well it understands things. And I'm continuously surprised because it wasn't that long ago that again, you'd have Tesseract or one of these other platforms that's very widely used and widely accepted. And even if you trained a model like Doc AI or any of these things, they were good, but they weren't that good. Like I couldn't just drop random shit and be like, hey, give me, give me the stuff. And then that's awesome.

Yeah. I remember zero days when we talked about dropping an invoice or a receipt and having it just turn up in your accounting system with accuracy. And this is 10, 15 years ago from memory. Back then it was a hard, complex problem. No, it's not. One of the things I think we've got to be really conscious of is this idea of blast radius. So what I used to always say in the data space, Ghost of Data Past, is I'll work on a marketing campaign because the data we're going to get is going to be crap. And therefore the results we're going to get are going to be okay for moving a lever, but they're not going to be accurate. But if I was working on pharmaceutical data for a drug trial, then it has to be different because the blast radius of getting that wrong is somebody dies. The blast radius of an incorrect marketing campaign is you spam somebody. And I think this is going to be the same with using AI against data services. The standard, you've heard about, probably heard about the one where a lawyer used AI for the thing for the judge. And yeah, and there was a use case in Australia where Deloitte's had to pay back half a million dollars to a government agency because they used AI to generate their very expensive review document and it hallucinated a whole lot of case studies. So I think we've got to be careful about where we use it, but I can see that the domain knowledge it has now from all the data and tribal knowledge it stole is useful, right? If the blast radius is acceptable, that actually it's good enough to look at that, apply some context, and it's cheaper and faster than a human doing it and the impact of it being non-deterministic and getting it wrong.

But on that note, and let's, we can tie this easily back to governance and among other things. I guess the world that I've come from is very easy in a lot of contexts to generate, I'll use the Deloitte example, cause it's fun to pick on them, to generate a report and just shove it out the door. Why do you do that? Fuck if I know, but it's very easy to do that. And this is my current complaint with vibe coding as well. Right? It's just that people hit the button, they say, ah, it's fucking magic. And then they shove it out the door. And so for the love of God, do you read your PRs? Do you edit your own writing? Please go back and read what came out of the random black box. Places I've worked, especially when we were at Palantir, amazing at shipping things fast. Right. But I was with a bunch of people. I won't say where, but like I was with a bunch of people. I'd have to kill you if I had told you where, but, and they were, phenomenal engineers, very good at understanding the problem, write code very fast. And I caught a couple of times where I'm like, guys, just read each other's stuff. Like it's not, it doesn't take, promise it doesn't take that long. And then we got there really quickly. It wasn't a big deal, but when they're again, all exceptional. So it was a fairly easy thing to do, but I know tons of people that are engineers that don't read their PRs or don't have automated checks in place or not linting, like all this. Do you use Grammarly if you write? Do you spell check shit? Like, For the love of God, if you're writing a paper that has citations, fucking check the citations. This is like basic stuff. And I think that people are getting so jazzed about the fact, thinking it's literal magic and just forgetting everything. They're like, what planet am I on? Hit the send button, go for it. And just, yeah, just take— you saved 90% of your time that you would have otherwise spent writing it. You can spend another couple minutes just making sure it didn't spit out garbage and I'll rant on this for 2 more seconds. If it was a person, if you hired a genius level person, let's assume that you hired somebody tomorrow to help you with your job that is a certified genius. Would you read their work after they generated it or would you just say, here, okay, cool, and just submit it to your boss or your customer? I don't give a fuck how smart you are. I'm reading what you put together, right? I need to know what it says. I need to know what I represent. I'm accountable. So why in any of these circumstances would you not check this stuff like that? It just, it's mind-blowing to me.

It's an interesting one because as we know, LLMs are non-deterministic. And so people go, how do you trust that it's doing the right analysis? And Juan Zucato had a great comment many years ago where he said, how do you trust a human? And I sat there and I was thinking, yeah, the number of times I've seen a data analyst come up with a number and nobody peer-reviewed it. And we trust it because a human wrote some code. And I suppose the code's deterministic. You can go see the code and run it. Time and time again and get the same response, right or wrong that response is, but that's not the point. The point was you trusted that number, made a business decision, and nobody peer-reviewed the process or the code. And I'll go back to one thing, you know, is definitely with AI vibe coding at the moment. If you want to see how dangerous it is, we're a Google platform, so I love Google. I love their platform. I love their technology. I hate their partnering and I hate their marketing because It's the worst in the world. But anyway, just go onto Reddit, onto the Google Cloud subreddit, and watch how many students are going and vibecoding with the Gemini API and pushing their code to a public Git repository and then getting whacked with a $3,000 to $30,000 bill within 2 days because their API key is publicly available and people are just scanning Git repos and grabbing those keys and slamming them. And somebody should run their eye over it. And it's like, you just watch the unintended consequences of this democratization. But let's go back to that image one, because it's interesting for me. So one of the things we did with one of our customers a while ago, they were moving from a legacy platform to a new data platform. And they had, I can't remember what it was, but something like 8,000 Cognos dashboard reports. So they built over 20 years.

And people watch every single one though, too.

They were all active when they asked, which ones can we get rid of? The answer was none. And they'd spend a couple of months with a small team of really good data analysts, BAs, trying to document them. So all they really wanted to understand was how big's the data estate, right? How many of these do we have? What do they look like? Which ones do we migrate, rebuild or migrate first? Which ones don't? And what we ended up doing is we ended up building a tool called Disco. Effectively, they exported out all the Cognos reports as XML. Yep, that was definitely a disco dance.

I'm dancing for anybody that can't see. That's right.

Yeah. I have a habit of making t-shirts, so we have a t-shirt for— Nice. 80, the Disco. People can buy it online if they're really keen. And so what we did was they basically pushed their XML files to us and then Disco went and scanned it and we did a whole bunch of prompt engineering based on some patterns to say, what's the data model underneath of them? What's the information product canvas? What action and outcome do we think has been taken? Right? So we generated all this context and then pushed that back into a database so they could query it. And that worked, right? There was some engineering we had to do because doing it for one XML file manually works. Do it for 1,000, repeatedly you have to loop through. But the blast radius was small, right? Because really what they wanted is insight. And then what happened was they came back and they said, we documented the reports that were copied. So this report looked like that report, but it had a new filter. And this report looked like that report and it had one more column. So where people would just clone the report. And that helped them de-juke. But they came back with an interesting question and they said, can you tell us which are reports and which are dashboards?

Oh, interesting.

Hmm. Okay. And there was some business reasons why they wanted to do that. So what we tested was them uploading screenshots of the reports. And what's interesting is, yeah, Gemini, and I think this is where back then we, this is pre-Pro 2.5. But even then it was good. It basically did what a human did and said, if I see a single table of data, it's a report. If I see multiple widgety objects, it's a dashboard. And it went through and categorized them. And I was like, holy shit, that just makes sense. And the other thing I've been doing is uploading the information product canvas as a screenshot. So building a canvas with a stakeholder, taking a screenshot of it and putting it into the LLMs and then saying, Give me the metrics, give me the business model, give me a physical model, a whole lot of questions around it. Whereas in the past, what I'd do is scrape out the text for that and put the text objects in. Whereas now I can just put an image in there and I get almost as good a response. Now the key is the blast radius for what I want to do is understanding. I want to understand quickly versus I'm not going to go tell it to build an information product and deploy it. But yeah, I cut out a whole lot of effort and it feels magical.

So I'm gonna, I'm gonna do two things. One, yeah, image-based LLM use is awesome. I saw someone recently note how we've been talking about RAG, but doing it purely on image data as opposed to on the text itself has been wildly more beneficial. And that's because there's subconscious cues there. There's things that we pick up on when you look at the layout of the text, what else is on the page, how the text is organized. And it's not just about looking at the text itself. And that's been hugely beneficial. So that's whenever I do data file processing today, extraction, structured, unstructured, that kind of stuff, it's all visual oriented. I try to avoid scraping entirely. Now I'll give you a kicker, which I don't think this is IP at all, but like a kicker is Excel. Can't PDF an Excel document. That's just extra stupid, right? We can't do images of Excel, but Excel is a wildly interesting thing. This is where we could get into structured, unstructured, right? Excel under the hood is really what XML or that weird format that it uses, right? In all, I would define that as semi-structured. Other people may fight me on this. That's fine. But I would define that as semi-structured. Because it has structure inherently, but it is also variable in nature. So I consider that separate. But those documents are hard to understand because hell, I've seen too many really shitty financial models that are just like 20 tables in one tab. And I'm like, oh, for the love of God, why did you do this? Who are you? And what kind of chaotic person? Organize your shit. Like, give me a break. Insane. Like, no one's scrolling around thousands of rows left or right. This is wild. When you've seen these, like people build financial models, just the most ass backwards ways on the planet, but visually interpreting them, assuming that you can get away without the pagination or anything like that is very good. It's way better from a visual standpoint. But I think that the big thing, and I'll circle it back to the main topic here. If you think about now, there are companies out there that sell data or that have a data-oriented process, right? And that is their main revenue driver. And then you think about shoving a large language model into that process in some capacity, mostly probably because it's unlocking some new features for you, or you're moving faster, you don't need humans, blah, blah, blah. In some ways, it's not really different than it was before. And I know that sounds really weird to say, but the reason I'm saying that is because if this is your product, you always had, whether you acknowledged it or not, You always had a need to set up proper process to ensure you have a quality product. So for me, when I think about large language models and their use in any of these processes, it's all process engineering. Yes, context engineering, blah, blah, blah, blah, blah. But like it's process engineering really that we're talking about. And especially once you start talking about multiple agents, I'm air quoting agents because that's a different bag of tricks. But once you get multiple autonomous processes interacting, like it's all process architecture, right? You're— I think the most common one that a lot of people talk about is the whole maker-checker paradigm of you make it, I check it. That's how it works forever and always. We don't cross lines. That works reasonably well. There's some sort of accountability structure and review system and so on and so forth. PRs have the same thing and people still put in shitloads of additional automation, but it's process structure, right? So even if my entire data product, just for hypothetical sake, is me just shoving a prompt in and hitting play repeatedly and then sending it to somebody, you should still have some sort of review system. You should still have some sort of checks to make sure it's not garbage. Every major manufacturing, et cetera, everybody has this. And that's why I like, I still think the lawyer and the Deloitte example, I'm sure the PowerPoint Deloitte put out was huge and it had had a billion references and was blessed by the Pope and shit. Like, I'm sure it had all this stuff, so it'd probably make it really hard, but we're data people, right? Rip all the fucking things off, go cross-validate them with a deterministic process, flag the ones that don't— like, you can bootstrap your probabilistic process with deterministic shit. It doesn't have to just be like everything's tossed to the wind, we're using a new tool, just Hail Mary and pray it makes money and VCs will pay you. I don't understand that mentality. We've been doing this for a long time. I am definitely the first one out the door to use AI and LLMs for things, but it doesn't mean I'm going to let it drive my car or care for my kids. I want some structure and controls around it. No different than a human, right? And I think it's a lot of people poo-poo the idea of treating LLMs as human, making them human-like. But I think that it's a very good analogy. It corroborates my feeling towards vibe coding, which is why in the fuck would you approach an engineer and just say, build me a website and walk away and think it's just going to work? Like they're going to build something.

It's even worse now though, because you read people saying my boss vibe coded over the weekend and gave me the code and told me to push it to production. Like there's a problem.

I have a feeling that's clickbait though.

Yeah, probably. Although I've met some managers. One of the questions I've got around that Deloitte thing though, was the first prompt in their agent always you telling it what the latest shape to use in the document. I like where pyramids this week, or where circles, or where matrixes, because you've got to change the shape of every document. Of course, that was sarcasm, by the way.

That's where all the money comes.

The interesting thing is this idea of Maker-Checker. Was that what it—

Maker-Checker? Yeah, it's a process paradigm.

Yeah, it's around factorization. It's about repeatability. It's about deterministic. And then we have artists, we have craftspeople that make things that are just one and done. They make it once and it is not deterministic, it is probabilistic. It is a piece of literature.

I'm going to fight you on this. I'm going to fight you on this. I agree with you that if I am painting a painting, that it is easy to think about I'm painting the painting and it's just, I paint it and it's over. But I don't know if you do any art, Shane, but I cook. I'll relate this to cooking. Do you cook?

Yes.

Do you taste your food while you're cooking?

Yes.

Good. That is a good thing you should do. It's not exactly make or checker, but if you have a partner or someone you're cooking for, sometimes you have them taste it. It doesn't mean you have to have them check it just at the end and you get to redo the whole fucking thing, but at least having some process that ensures you are not going off the rails entirely. I make a lot of random stuff. A lot of stuff I make, I've never made before, just because it's fun. And I always taste it midway through because you never know what might go wrong, or you might— there's little tweaks, more salty, more spicy, too spicy, that kind of stuff. But the maker-checker paradigm, that is a very particular paradigm. There are probably some ways that that pattern is implemented where it's you finish your thing, give it to me, I review, say whether you suck or not, hand it back to you. There might also be versions of that where it's midpoints, right?

Like, okay, so let's take that maker checker idea and say that the process that even an artist, a craftsperson checks it themselves, right? They may have another person that's as experienced as them or what, but they are checking, right? They are always checking their work.

Hopefully to some degree.

And within GenAI, I have 3 types. I have what I call ask AI, which is where you ask it a question, you get back a response, you ask a question and you're chatting with and then you go off and make the decision as a human, right? And ideally get another human to check your work. Assisted AI is where it's watching what you're doing and it's going, hmm, based on what I know, you might want to think about this. So it's prompting you, you can listen to it, you can ignore it, but you carry on, you finish that task. And automated AI is where the machine does it and no human's ever involved. Yeah, it just happens. And so if we take that idea of PDFs and if we think about code being deterministic, deterministic and LLMs being probabilistic. And we think about if I wanted to just upload a PDF and get some tribal knowledge back about it, that is a probabilistic problem. I can put it up there. I'll get some stuff. I'm in the loop. I'm going to make some decisions. So therefore, it's an ask kind of feature and the blast radius of me getting it wrong lives with me and my maker-checker paradigm. If I was automating that PDF. To go into my finance system and put in the number, then maybe I move to an assisted model. I upload it, the machine identifies all the fields for me. It comes back and goes, this is the invoice number. This is the tax amount. This is the total amount. This is the supplier. You happy? Click go. So it's a system. It's automating all that drossy shit, but I'm still making the final call of, yeah, that's right.

Correct.

Versus if I take it to fully automated. That's when I'm dropping in 1,000 invoices, they're going into my finance system and a payment has been made and I am not in that loop. At that stage, in my head, I go back to code. I go back to deterministic code that is looking at specific places on the layout of that invoice and saying, that is the number. And if there's no number there, don't take it from anywhere else. And so I would say. At the moment, I would not use an automated GenAI solution in that use case. But that's only because I haven't tried it lately. Like you said, when I uploaded images 2 years ago, it sucked. I upload them a year ago, it's got amazing. I haven't done it this week, it's probably going to freak me out. So where would you sit, right? When do you jump from assisted human in the loop, Maker Checker, to let the bloody thing take this unstructured or semi-structured data and just human out of the loop.

I'll say a couple things. One, there's the very obvious part that I'm gonna state because everybody's gonna say crap about this, but security. Obviously there's a security element to this. We're talking about financial statements, blah blah blah. Let's remove that just for argument's sake. So I'm going to repeat again, we're removing the security concern here and the data privacy and all that shit just to have a hypothetical conversation. Before people are like, privacy, blah, blah, blah.

Hold on, what's your definition of security?

Ask Joe, go call, I'm gonna give you Joe Reese's phone number, you can call him, he loves to debate semantics. Anyways, this is gonna sound funny. I don't know where that line is, and I am actively and consistently trying to do it the automated way as much as possible. And I often equate a lot of these things to like meditation, right? This is all about building good practices. You have to set up the conventions in your mind, build the muscle memory to do the thing that you may not have done before. I'm with you. It's very easy for us as engineers to think about, hey, I want to rip this one cell. I know it's in the same place every time off this document, write the code to do it, right? And don't get me wrong, there's an over-engineering element to this of throwing an LLM at a problem like that is definitely a bazooka at an anthill, like 100%. There's also a time to market, I'll call it, component of this, which is how fast can you write that code compared to how fast I can go to a website and upload a file and ask a question. I'll quick draw with you and I'm willing to bet that I'll win and I'll still get the same answer, right? And then there's a middle ground if you really want to fuck around, which is have the LLM write the code to do the deterministic thing. That's a whole nother, like, I don't have to have the LLM just do the work. I can have the LLM write the process to do the work. And then you get a little bit of, a little bit of both because it's a deterministic process that was generated very fast. This whole name of the game is speed. How fast can I do X activity? That's our North Star. If we're talking about expense parsing, I literally just did this the other day, right? I dropped a PDF onto a platform and it read in the expense. Cool. Did I validate it? I didn't. Actually, that's not entirely true. I knew what the number was before I dropped it in and it happened to produce the right number and I was like, cool. So that's my pseudo maker checker. Most of these platforms these days, and you made a comment before about trust and trust in determinism, trust in code, right? Especially when we think about people, a lot of that trust is just based on transparency, transparency, auditability, the ability to go back. And this is a very common paradigm in data. Like, how do I roll something back? How do I undo something? We were talking about this yesterday or the other day, right? Especially from a version control has given us this wonderful sense of security. I can go back. I know what happened, blah, blah, blah, blah, blah. I can yell at Shane because Shane fucked it up. So we have blames. I think that in this world where an LLM can do the work for you, again, from a cutting down time perspective, it does cut down the time. Maybe I put the document side by side with it. It was very common of the old platforms and the new ones. Great. Look at the document. Here's the value. Here's where the value came from. Now, looking at a form, let's just make this more complicated because it's fun. If it's a financial statement, and there are a bunch of companies out there that just straight do this, this is their business, right? Talking about AI data services. Their whole job is to take documents and PowerPoints. I won't name names, but documents from investment funds as an example, and pull off the values. Now there's definitely an intelligent person out there saying, why the fuck don't they just put the values out in an API? And that's a great question, but That would be logical and God knows none of this shit makes any sense. You know, a lot of them put out these documents and they probably got fucking pictures on it and all kinds of stuff. And they're in hell if they're all the same. They're definitely all different because why would they be the same? That would make sense. There's whole businesses centered around just ripping these documents, doing the OCR, whatever. It's one thing if it's a table and this kind of goes to the facts at Bloomberg stuff too. If it's a table and you're just like, I always want cell 1, column 1, row 1, give me the number every time. Not super complicated, not a lot of assumptions that need to be made. And I think assumptions is one of the big things I think about when I think about LLMs and probabilistic patterns and things like that. And I can give you my convention there, but the number of assumptions it needs to make also relates to how much context you give it, how clearly you can describe the things it needs to do. So my usual grid of this is On one side, it's the number of decisions that you need to make. And then the other side is how much information you've given it. That's any process. It's straight up any process. It's for a human or machine or anything at all. I could say, Shane, go make me a cake. I've given you no information. You have to make shitloads of assumption. You could make a cake out of mud and theoretically you've delivered. You've given me a cake or you could have made me a Lego cake for all I care. That will also suffice. But that doesn't necessarily meet meet what I have for expectations. So again, long-winded piece here, but if we did something more complex from a document processing perspective, a, hey, give me the revenue and the revenue's in the cell, but then there's 4 adjustments and footnotes and 12 other little things that you need to take into consideration. That's how it gets complicated real fast. And we know in financial statements, this happens all the time. Like accounting for a lot of people seems like it's a very rout thing and is actually way more creative than you realize. So yeah, that's where this stuff gets creative. So to my process and doing things like this, I always start small tasks. I always try to automate it if I can, if I'm comfortable with the security, et cetera. I always try. I have to start there to understand the bounds of what can be done or not done. And sometimes it's process architecture to repeat that, or sometimes it's how much information am I clearly articulating? The instructions, which is prompt engineering for lack of a better term, which is for humans just fucking communication, which is always comical to me. I think that there's no bounds in some respect. You should always be trying to see what it can do. Again, I don't think we've fully adapted to understanding that I can use this every day. And that's why I think it's helpful for a lot of us to think of them as humans because it's just, oh, if I had an assistant, You'd know instantly what you'd have it do, short of pick up my laundry, which it can't do.

I think there's, there's an interesting kind of thread in there that I've been thinking about, and I'll just unpick it. When we build our data platform, because we've been building data platforms for years as consultants, there were a bunch of core patterns that we knew were valuable. And because we pay for the cloud infrastructure and cost, not our customer, we are hyper-focused on cost of that Google Cloud stuff, cuz it's our money. We log everything. Every piece of code that runs, we log the code that was run, how long it took. Yeah, we have this basic big piece of logging where we can go back and ask questions. Yeah. Are we seeing a spike on the service? Which customer, you know, or customers? Is it one customer? Is it a volume problem? Is it a seasonal thing? Is that across all customers and Google are changing their pricing model, which they do on a regular basis. And so that's just baked into our DNA. Yeah. Log the shit out of everything. Cause at some stage we're going to have to go and ask a question of those logs. And we need that data. When we started moving into AI-ing with our Agent 80, we logged some stuff, but not everything. And as we saw our partners start to use her for really interesting use cases, they started saying, here's some source data, what kind of data model should I start modeling out? Cause we haven't seen it before. They have a piece of data and they want to transform it. And so they saying, what we call transformations, we call them change rules. What should the change rule look like? Talk me through how to create it. And we just, we saw those questions happening because we were logging the questions and we started doing more context engineering to give her better access to things that gave her better answers. And what we didn't do was when we moved to the, and air quotes here, reasoning models, we didn't ask her to log the reasoning path. Every time we asked her a question ourselves, when we were testing something, we would ask a second question straight away of how did you get there? Because we want to understand. Where she's grabbing the context, the prompt from, because that's what we wanted to tweak. And what was interesting is our principle was log everything up the kazoo, and yet we move into this AI world and we didn't do it. And of course, once we saw our partners asking questions and we're like, how the hell did she get that answer? Because it's not the one we wanted her to have. And then we ask her the same question and we get a completely different answer. We're like, okay. And so then what do we do? We just put into the prompt or into the framework effectively, log the path for every question you answer. And now we have a richness. So that's the first thing. It's really weird how as data professionals, we have this baked-in principles and patterns for our entire life. And then as soon as we move into this AI world, somehow we forget what we always do. The second point's around complexity. And so one of the things I do when I'm coaching teams is I will ask them to draw me a map. Draw me a map of your architecture, draw me a map of your workflow, draw me a map of your data sources. Like, just— Draw me a map because I'm a visual person and it's really what you say about giving an LLM a layout, a map, and distance between things, similarity or clustering of things. They have visual indicators as humans we're really good at using. And so when I'm working with teams, the reason I want a map is the number of nodes and links, the number of boxes, the size of the map will give me an instant identification or understanding of complexity. You have 115 source systems there that are going through 5,000 dbt pipelines. I'm going to go, that is a complex problem. You show me your team structure and it's got 4 layers of teams all handing off to each other and they're in pods and squads and there's 150 of them. You've got a complex business organization team topology. And so what's interesting, because I was just thinking about as you're talking about it, is If I take those reasoning paths and I basically do some simple statistics and I cluster two things, how many steps did it take? That will infer the complexity of the context it's trying to use and the task it's trying to do. And the second one is clustering around reusable paths where it's constantly doing the same thing. Means that is almost a deterministic behavior because it's constantly going through the same path. And when we see an edge case, an outlier, where it's gone through a completely different path, we're like, oh, is that because different question? You just went and hallucinated for some weird-ass reason. Or, you know, there's something interesting there because it's different. And we know different has value, which comes back to one of the things around eval. So there's lots of work and it's a new hot thing in the market is how do I eval my algorithm? Yeah. We used to call it test, right? And we know what data people like testing. Go back to my point of principles and patterns that we apply for our data work every day. And then in the AI world, we don't. Let's talk about the ones we never apply. And now you get this whole idea of judging. So the idea of, if I ask the LLM the same question 4 times, and then I go and determine whether the answers are similar, and if they are the same or very close, then I should have more trust in the determinism capability of that LLM.

Answer.

And that's a presumably, I haven't actually bothered to go and see any research papers to say if that's true or not. But is that what you do? Do you tend to use that as an eval process when you're doing all this work to apply AI to reduce your effort?

Yeah, I always try and set up test harnesses of whatever kind. And I put evals in that same bucket. I agree. And in our book club, we're reading AI Engineering by Chip, which is a great book, talking all about the different evaluation methods and types, et cetera. I agree with the fact that I think this is an under-invested area. I think it is an extraordinarily important area, and I equate it to testing in code. I think that there's a ton of code out there that isn't tested for whatever ridiculous reason. It's always people want to move fast, there's businesses, blah, blah, blah. But even if that means ad hoc testing, which is what you're noting right now, run it a couple of times and see the answer. That's better than nothing. I've done a variety of methods now. I do LLM as a judge. I'm still back and forth on this. So the short of this, if you're unfamiliar for anybody else, it's effectively kind of like an adversarial. I have a system that generates something and then I have another LLM that more or less evaluates it. I've done it with different LLMs evaluating others. So I might have Gemini as my model and I'll use Claude as my evaluator or or multiple evaluators, things like that, or have it ask different questions from an evaluation standpoint. I've done that a variety of different ways. That's been very useful. I think anything where you can run your tests at scale— scale might be, that's a very overloaded term— anything where you can run a lot of tests, not spot tests, a lot of tests and a lot of extreme tests. I always tell people the QA joke about the bar. You know that joke, right?

I may do, but tell it in case.

I'll tell it anyways. I'm going to butcher it, but the joke goes something like, somebody builds a bar, QA tester tests the bar, orders 1 beer, order 99 beers, order a million beers, everything works fine. First guy in the door says, where's the bathroom? Bar explodes. Great joke. Very appropriate in this circumstance, right? Of you don't know what people are going to ask. And I understand that large language models obviously are an NLP thing. It works on natural language, blah, blah, blah. There's many things in that case that are open-ended, so to say. And I think that the chat paradigm, especially as a UX paradigm, introduces a wild world of open-endedness that anything can happen. And I talk to people that are building, again, AI products, and I'm like, do you want me to come to your product and ask you what my favorite color is? Because what's your thing going to do then? You may have built this wonderful chatbot for finance or something, and I'm going to walk up to it and be like, How do I make a chocolate soufflé? And it might give me a wrong answer. I might be like, this AI is terrible. I walked up to my finance AI and asked it where to drive to for lunch and it didn't work. I was like, no shit. But there's a piece of that, which is like, it's not intended to work. So the open-endedness is pretty wild. But I do appreciate there's a bit of this, which is like almost chaos engineering, which is I just want to let it like just blast it with like random stuff. I also think that having a prebuilt dataset, you should have some tests, almost like unit tests to some degree. Ask it a question, you should get an answer. Then the next problem to your point is how do I evaluate whether the answer is correct or not? And that, that is hard in and of itself, but there are conventions, ways you look for certain keywords, values, things like that. So I do think that evaluation is massively important. It's very easy to spot check a couple of questions and just be like, yeah, it's good and move on with your life. And then you're handing people free cars because they fucked with your chatbot. So it's massively important. And I think that the more we can, in a manner of speaking, beat the shit out of the machine and like literally test it to the nth degree, even if you are using a language model, I think that's helpful because the language model is going to come up with shit you probably didn't even think of, like jack the temperature up and just tell it, ask random questions and make sure it doesn't ask the same question twice for all I care. And then you, this is our maker checker a little bit. That can go review and say, hey, make any fucking sense, or did it just spit out random garbage? I think all that's super important. Like, evaluation, I think, is very understated right now. I think people are catching on a little bit, but you said it before, it's very comical right now that we've got this new toy and everybody just forgets who the fuck they are and where they are, and they're just like, yeah, cool, let's do it. You're— yeah, you're a seasoned professional. What the fuck are you doing?

In an organization, I was talking to somebody the other day, and they're in a large organization where security is key. And then they were saying that their cloud provider, somehow the LLM part of the platform got turned on for the whole organization.

Ooh, fun.

Wow. And so it's interesting how this AI thing seems to change behavior. People—

I agree with that. I think that case might be a mistake, just like a human error.

Yeah, maybe, but you do see organizations that care about security and governance and privacy, and then you see people in those organizations grabbing things and putting them in their personal LLM because it's so easy now to take a screenshot. And yes, they're doing the wrong thing, but it's amazing how human behavior is so different. When we talk about evals, we've got to go back to complexity. Let's just go back to data. If I have a piece of data and I have a piece of code, and I have an assumption or an assertion. I can define what the assertion is. I can run that code against that data and the data doesn't change. The code doesn't change. And it can tell me whether that assertion's right or wrong. That's 3 moving parts in that test suite. Why we don't do that a lot, it really intrigues me. Now let's talk about that within GenAI. Oh, actually, no, hold on. I've got the data or the thing that I'm using, right? The input, the PDF piece of data. I've got I've got the question that I ask it, which is effectively the code to a degree, and I've got my answer, my assertion that I expect. Ah, but actually I've got an LLM model, right, which may or may not have been updated by the vendor without me knowing about it. Ah, I've got a prompt. I've got a piece of text that's actually embedded in that process that may or may not have changed or be interpreted differently. I've got RAG or context. I'm pushing it some other stuff that may or may not have changed. Ah, the PDF is exactly the same except the dollar value for the invoice moved down 25 pixels. Now that's not going to make a difference, or does it? This is where we get to, is that actually now if you do nodes and links for the process, we have lots more of them and therefore the complexity of what we're testing, the things that can change is massive. And talking about change, again, one of the human natures I've found is once I start using a model or a tool, I get stuck. I use ChatGPT a lot for writing because that's what I started out with. I use Perplexity for searching now over Google. So I Perplexity now, I don't Google. We use Gemini for our platform because we're bound to the Google Cloud. I use Claude a little bit for MCP testing, but I don't use it a lot because I don't code. And I'm sitting there going, I can't remember the last time I actually tried to decide when I would work out whether there are better models for what I want to do for the thing that I use ChatGPT for. So how do you deal with that? Given the models are changing all the time and that the models tend to be good at specific tasks. How often do you reevaluate the tool or model that you're using in the work that you do?

Not as much as I think I should, but I evaluate the new ones as they come out. So when GPT-5 came out, like I reactivated my OpenAI subscription. I really don't like OpenAI. I don't know what it is. I maybe they, I just like feel like they're like evil empire or something. I, like everybody else, started using their products when they came to market and even before chat days, and I'm not going to be that guy that's just, oh, I've been doing this before. But I used that stuff before and it was interesting and I was very curious about it, especially given like the work that I do. When ChatGPT came out, I was definitely all over it and I was very fond of it. So I will tell you that I very often use for understanding information, I use Gemini. I appreciate that one. I am a Claude fucking junkie when it comes especially to coding. And I'm a heavy cursor user, like very heavy cursor user. And I will tell you my favorite thing that has come out, period, that— no offense to you, Shane, but I have definitely done some work while we've been talking, right? And this is my favorite thing. I have my ticketing system wired up to background agents or cursor. So like I write reasonably verbose tickets that explain exactly what I wanted to do. And for context, like the scope of these tickets is like, hey, You were taking in parameters 1, 2, and 3. I don't want that. I want it as one parameter that looks like this, and then I want you to do this with that. That's the scope of tickets that I'm writing because I want to be in control of it. And it's like a micromanaging approach. I write the ticket, I say go, and it writes it and I just keep on with my day. So I could be like walking my kid or my dog or somewhere and I will just fire fucking tickets off. I'll be working all the time. It's amazing. That's my favorite feature by like a Absolute mile and a half. And I love Claude 4.5. I think it's amazing. And the code quality is phenomenal. And I'm probably paying them a disgusting amount of money, but it's, I think it's totally worth it right now because I can move epically fast, multitask. So Gemini for information, Claude for code, OpenAI for nothing, unless I like just want an alternative opinion. And I definitely consider them all like different people to some degree. I very much think of them like people. They all have their different quirks. Some things are good. Some things are bad. Some things are like, especially from a code perspective, some of the models are more overeager than others. Some of them want to try and cover more edge cases than others. And sometimes I'm just like, no, just fucking do what I told you to do. I don't need you to do 20 other things. And I have all kinds of rules set up. So I have it set up in a way that it will do the extra things I told it to do and not the things I don't want to do to some degree, which is very nice. I'll make a separate point. And I've always been this way. For better or for worse, but a healthy degree of paranoia, I think, is always good as data professionals, like healthy paranoia. There is unhealthy paranoia, but healthy paranoia, I am a big believer in. And I've met many people on my career and I think a lot of us agree with that level of just distrust, inherent distrust. And it's not to say that we don't trust everybody. I actually think I'm a reasonably trusting person, which I'm sure someone could take advantage of me just hearing that, but I still look up the things that I ask the LLM to do, to put this into context. I still cross-check things when I write code with LLMs. I read all of it, like all of it. I see my role as I am the reviewer, I am the checker. I always joke with people, it writes faster than me, that's why I use it. It types faster than me, but I am always reviewing what it did. If it does extra things, great, brownie points, take some extra credits so you can whatever. I'm very content with that, but I I absolutely read everything as much as humanly possible. And sure, I do random experiments where I just tell it, build me some shit and I don't really care. And that's a fun experiment. But I've worked on a project recently that I'll tell you about where, without going into too much detail, the project wrote something in a language I'm less familiar with. And I read some of it and I definitely spent some time, you know, I used an LLM to do this and teach me about the other libraries, teach me about the other things that I'm just not as familiar with. So I spent some time doing that in different LLMs, different context windows. So like, hopefully there's some, I call it arb, that's happening there where I'm hopefully getting the right information. I also do Google things still. I don't Perplexity things. I think I Perplexity things just sometimes, cause I want to see what the answer is, but I do still Google shit. Cause I'm not going to trust it unless it's like, if it, if I asked it to write something in Beam, I better I'm going to go to the fucking Beam website and check to see that it did it right. And especially from a code perspective, and they may have fixed this, but my favorite like red flag reading somebody else's generated code, one of my favorite red flags, there's plenty of them, the comments and all this other shit. My favorite one for a long time was that especially when you had it generate an AI model or something that used a language model, it usually wasn't trained on enough information to know what the latest model was. So it was always a really big dead giveaway when you generated code to use like Gemini and it puts 1.5 in there and I'm like, you didn't read this. I know you didn't read this because it's using a model that we all know is a year or two old now, but you didn't spend any time just looking like that's a really basic thing to look at. And you didn't even look at it. I usually demerits for that person. That's a very glaring thing for me. Because just read the code. It's not that hard. I promise. It's not that even if you skim it, it's not that bad. It's better to do that. So to the point of, do I evaluate the tools? Do I do that? Yeah, I absolutely still do. I definitely still spend time learning about the stuff outside of my use of it as well. So that way, again, same as a person, even if you meet the smartest person on the planet, I still want to have my own authority. I still want to fact-check people. This, maybe there's a very existential philosophical comment here about between fake news and the internet and the web. Sorry, Juan, the web. So I'm a big proponent of cross-validating things. I just believe that. And sure, not everybody has time to do shit like that. I definitely don't do it for everything. There's some things I just take face value, but I do believe in that. And I have, I have a little kid right now and I'm in the phase of why. Which I love. I really love this phase. I'm sorry. I think my wife hates it, but I love it because he asks why. He's not at the annoying why phase, but he asks why for a lot of things. And I take it as an opportunity to go look them up. And we talk about things. And if he asks a subsequent why, I'll look that up too. I don't give a fuck. I love looking things up. But you should feel more empowered to do so. And I feel like people just are short attention span. They like just don't fucking care.

I think you're right. I think that idea of trust but verify and coming back to your Gemini 1.5, they probably don't know there is a Gemini 2.5 because they haven't done the work. They haven't done the reps to become an expert or any experienced in it to know that actually there's a difference between 2.5 Pro and 2.5 Flash.

Yep. Or Flashlight or there's Flash with dates after them. Like there's all different fucking shit. Yeah.

So just want to look at times. You want to close out with 2 questions for you. So right at the beginning, we were talking about the lawyer stuff and expertise and tribal knowledge. And given that all the LLMs have gone and stolen tribal knowledge that was publicly available and often privately available, do you reckon the world's going to move that humans who have tribal knowledge are going to stop documenting it in a way that it can be found? Because actually the value now is trying knowledge?

I want to say no from like a humanity standpoint, but I do think that— I think the world will become a lot more polarized. And I'm going to explain this in a weird way, and I'm not intending it. I know that there's a political backdrop and everything as we're recording this, but if the model inherently gravitates towards the mean, always gravitates towards stereotypes, that kind of thing, and we're having a feedback loop to ourselves on this, our own opinions are going to be potentially squeezed, like kurtosis and all that fun stuff, if you want to get into that. Do I think that people will stop? I think people will use these tools in more automated fashions, and then that will create its own probably problematic feedback loop. I do, however, think that the pendulum will inevitably swing the other way, where I think people will still use these tools, still use tech to create content that maybe they didn't record before. I don't have a Substack or any of this stuff. Yes, I am talking to somebody about recording a podcast because why the fuck not, even though we're on right now. It's mostly from, I want to help their business kind of thing. But I think that hopefully technology will lower the barriers of entry to recording more. I hope that inevitably forces the pendulum to swing the other way at some point in time. And I remember reading an article recently, something about someone saying how there's no second internet. There's no second web, really. Again, I apologize. I like Juan's perspective on this. It is the web, not the internet. And I make actually a mental note, like always to correct myself. So thank you, Juan, for that. But there's no second web. Like we've trained these things on so much information. At some point in time, I do think we'll start recording more. And then there's a very interesting dynamic of, okay, now everything's recorded to your point. Tribal knowledge at all, what's left? I'd like, and maybe this is overly optimistic and potentially naive, but I'd like to think that if we continue to use these tools at whatever capacity, we'll advance our knowledge faster and therefore there will be more knowledge and more interesting ideas and creative ideas. And I'll emphasize something different that is related to this question, which is I am a huge proponent of the fact that I think that especially as the pendulum swings back and forth, I am very bullish on the arts. To your point of painting earlier, art is inherently creative. It's inherently random. It's inherently wild. There's no predicting that. You can generate pictures and stuff. We all know how shitty those come out. And some of them come out great. But for better or for worse, art is something I don't think a lot of us understand. We still don't understand how we work. And I think that in comparison to science and science, I apologize to anybody that's in science. I'm not trying to demean your field, but I think that I'm going to call sciences things that are out in some nature. They have predictive process and results, so on and so forth. So I'm very bullish on art and I think the sciences, quote unquote, are going to be marginalized away. And that's part of this whole cycle in knowledge, et cetera, et cetera, as it learns more. But I do hope that. Technology encourages more people to maybe just do random shit, write down more, write down random ideas, write down everything. I don't think that we've gotten to a point where all thought has been explored, right? That's what we're talking about to some degree, where all possibilities of thought are captured. Have you seen Jepson's stuff?

No.

Look, okay, I will plug Vioxx right now because I— it's one of those things that you watch it and you're like I'm like, wow, like this is cool because he's trying to replicate like evolution and evolutionary thought as like part of his process. And it's like a, I'm sorry, Jepson, if you're hearing this, but it's like a very sophisticated Monte Carlo in some respects. It's try everything or random forest, whatever you want to say. And it's way more sophisticated than that. I'm generalizing here, oversimplifying, but I do think that there's something that we have that the machine doesn't yet. And it's creativity and that's art. And I think that's, I hope that there's a pendulum swing and don't get me wrong. I think that the vast majority of society is going to gravitate towards the mean. It's going to be some massive enshittification of fucking everything. But I do think that for people that are creative thinkers, like that, this is the time.

I agree. I think about data products without defining them or defining the definition of defining and I've reviewed 2 or 3 that are currently being drafted around that. Some are what I would call bringing product thinking to data, and some are around a thing called data product. That's how I categorize them after reading the draft content. I know a couple of other people that are starting to write something around product thinking with data or data products as a thing. And my view is good. The fact that there'll be 5 or 10 books in a similar subject space, the way they're telling the stories is different.

Different.

The storytelling is different for every book, even though it is relatively in the same domain or subject.

Yep.

And that's great because what that means is they can write them because writing is easier now and they can publish them because publishing is easier now. And then I can read them and I always pick up something new, something I didn't know, something that entertains me or educates me. And then I assimilate that into how I think. So I'm with you, push more out. But I think we will see some people try to put paywalls up because that's the tribal knowledge is how they make money. Interesting times. I think the other one, and we're out of time for this one, but I think given the fact that tokens are subsidized so heavily at the moment, and we use the AI tools or the GenAI tools to automate stuff for us because it is faster and it is cheap enough that we don't have to care. It's going to change. And when we get hit with the true cost of those things, then people who have built good automation, who have built things that are efficient and optimized and aren't lazy processes or lazy code, they're going to be better than the people that just put 1,000 PDFs in there with no context, no prompts, and get an answer. Probably your product will be one of the many AI startups that die 1,000 deaths.

Or maybe gets bought. Yeah.

And then they say, yeah, there is a difference between being bought and being acquihired.

That's valid. Valid. Very valid.

So yeah, again, what's your definition of buying? Or what's your definition of different mining?

It's when you put flame to the—

yeah. Just to close it out, people want to find you and find what you're thinking. You've already said you don't have a Substack. You might have a podcast.

Substack. LinkedIn's the easiest way to find me.

LinkedIn and Practical Data Discord because you're correct. One of the more active people in the community. That's how we met. Come and join us. It's free.

It's a great, it's a great community. I always advertise it to people. I'm like, just, yeah, who do you talk to about your business? Come talk to us. Yeah.

And we will talk back.

Sometimes they might just tell you to fuck off, but that's fine. It's also cool. We're very unhinged.

You might get a meme. You're going to get a meme.

We are a very unhinged group.

Thanks for that. That's been an interesting chat. We kind of went all over the place, but it's good to talk to somebody that's actually using AI to build data services.

In production.

and production and monetizing it as much as possible. Because that is an art or a science, one of the two.

All art. All right.

Excellent. I hope everybody has a simply magical day. And that, Data Magicians, was another Agile Data Podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to agiledata.io.