Redesigning traditional data systems to support Large Language Models and AI agents with Mayowa Oludoyi - Episode #80

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile Twitter products and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, hey, that's not what I wanted, and then you hear your team reply with but that's what you ask for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the Green Book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hi, this is Mayowa joining from Nigeria.

Thanks for coming on the show. Today we're going to talk about redesigning current data Systems to Power LLMs and agents. But before we do that, why don't you give the audience a bit of background about yourself.

Thank you once again. Again, let me just say thank you for bringing me to the podcast. And like I mentioned earlier, my name is Mayowa. I started off my career working as a business intelligence analyst and honestly I still feel it's still one of the most interesting roles that anybody can get involved in. It's very interesting. And while working as a business intelligence analyst, this kind of gave me the opportunity to work on several projects. I started off working in consulting and so that kind of gave me the opportunity to work on several projects. We have clients working in telecommunication, in banking, in health, tech. And so starting off that way gave me enough opportunity to experience different projects. And while working I had the opportunity of working on several projects like working on data warehousing, building data reporting, power, bi tableau, and all of those interesting data projects. But then as time goes on, I move on to a new company after two years and then in this new company, you know, it's just basically a payment company that is in fintech. And then while I was there I did a lot of work around analytics, data engineering. And that was where my interest in data engineering started evolving. And from there I left the company after two years and then I moved into a bank. And then this time around I started working as a data engineer. And while working as a data engineer as a bank we have a lot of legacy system. There's a lot of things, a lot of moving part when it comes to data engineering in a bank, you talk about privacy, you talk about building system that are highly protected, all of those things. So while in the bank I started having this interest in learning how to actually build robust Pipelines. So I wanted to move from the normal building etl. And so I started learning and that was how my career has spiral all through the years. Yeah, here we are today, that move.

From being a business intelligence analyst, focusing on the reporting tabular power bi I'm assuming the data was served to you. So you grab the data you needed and you focused on visualization and user experience and then moving back into the engineering side, into the code that transforms the data, collects it all that kind of gnarly stuff, that's a common path for people. How did you find it? How did you find changing from a set of tools and a certain set of skills to then expanding them out into new tools and new skills? Was that an easy journey or did you find that step change quite difficult?

I'll say. For me it was not an easy journey. The reason being that the resources are there. Online most time the resources are not what translates to business value. So sometimes you have to struggle to get things done. So it was not an easy journey. But one thing that I feel really prompted this shift in me was that at the end of the day, I discovered that one of the most important process still remains the data. You can have beautiful dashboard, you can have machine learning models that are doing fantastic, but everything still comes back to the data. So I started asking myself the question. I think I spent a lot of time transforming data, cleaning data, whether to produce reports or to do machine learning. And so I feel like it's important to spend a lot of time around the data itself than even the product. I mean the end product is really important. Right, because that is the whole point. But then I feel like it's important to spend more time on the data. So that was what prompted. But then I had to look for a way to attend conferences, take courses online. And then that shift was not easy. But gradually I started building all the necessary experience that I need.

Yeah, and so often I'll see people that make that change from being a bi centric set of skills to that data engineering set of skills. Often they're in an organization that actually encourages that change. So they're in an organization, they've got a bunch of mentors, a bunch of people who've done it before, and they can get a lot of help and mentoring about the things they need to learn and a lot of feedback. In some organizations that's not possible. So people will go and study on the site, they will go and find courses and they'll try and learn it to help them then effectively change organizations to Change roles. Which way did you go? Did you find that you had mentoring and support within an organization to do the step change, or did you. You have to learn outside the organization you're in and then change jobs to go into the new role.

Honestly, I couldn't find a mentor. I had to take the other route. I went all out to get information. I have a couple of people who tell me exactly, oh, this is what you need to do. You can take this course. I have those people, I have people tell me that. But I didn't have the opportunity of having someone who will hold my hand through the rubies and show me everything I need to know. So most of the time I had to go out there myself, pick up courses, pick up books to read, and that was it. So that was the part that I took. But I understand what you're saying. There are a lot of people who have the opportunity of having senior data engineer on the team who can tell them what to do and then they learn through working on different tasks, different projects along the way. But I didn't have that opportunity.

And if we think about the idea of a curriculum, an idea of these types of courses, these types of tools, these types of skills, and learning them potentially in this order makes sense. So almost like a manifest or a curriculum of what you might, might want to do, rather than having to try and find it all yourself. Did you find that, did you find that there were resources out there that gave you the idea of a boot camp? Or did you have to go and talk to lots of people and cobble it together to yourself to figure out how you went through that path?

Basically, my knowledge was a little bit scattered at the very start because there was no tailored curriculum to give me a roadmap. These days I see a lot of roadmaps here and there, and I wish I had these when I was starting. Right. But then when I start, when I started, I didn't have this roadmap. I had to learn. So at the end of the day, I have a lot of scattered knowledge and it took a lot of time, a lot of learning to put everything together to make sense, actually. But I think that problem has been solved. I think I've seen a lot of resources online, a lot of roadmaps that I think is doing justice to that now. Yeah.

And those roadmaps, are they primarily technical and tool based, or are they about the process and the ways of working as much as they are about the technologies? Because again, what I see if I look at job ads or I Look at what people say they do. I'll hear dbt, spark, databricks. I hear tools and technology versus the skills and the tasks that you need technology to deliver them. But they're just as important as the technology itself. So what did you find?

I can share this with you at the end of the session because I don't want to mention names. So there's this particular roadmap that I've seen which I think is very good. When you look at the roadmap, they didn't spend a lot of time talking about tools. Tools can change, but they spend a lot of time around processes, which I think is really great things, fundamentals that I think everybody should know when it comes to data engineering and even all that domain I think is really good and shared the same opinion. I see a lot of roadmaps where they talk about Spark, dbt. I don't think that is the best. I think it's because tools will always change. Tools are like, is a means to an end. It's not an end in itself. So I feel like roadmap that talks about the real process, the real principles. I think they're very great and I think that is what we help a lot of people to move up quickly.

We meet each other on the practical data community and it's still one of my favorite, most active communities I'm in. And yeah, I think something we should look to do is creating that roadmap or that curriculum as an open source roadmap and curriculum as part of that community because lots of people have that struggle when they're trying to get into the domain or jump across the roles. Where do you start? And so last question before we get on to the current data stuff. This whole onboarding of a person into a new career really intrigues me. So when you talk about if you had a founder mentor who would have helped you through that process, what would you have expected them to do with you? What would you want them to help with?

Sonic, are you repeat that again.

So you talked about the fact that you were looking for a mentor and you struggled to find.

Yeah.

So you had found one. Let's say somebody had said, yep, happy to help you take that next step in your journey. What would have been valuable for that mentor to do? What would you expected them to actually do to help you progress your career?

I think one major thing I look out for in a mentor is not to teach me maybe how to write code, but rather just to show me exactly what I need to succeed. And I will explain What I mean, so for example, when I was trying to break into data engineering, it was difficult to actually see somebody who have the experience of what happens in data engineering. There is a lot of resources on how to write SQL, Python and all those things. Right. But I need somebody who work in data engineering who tells me this is the kind of problem we solve. This is why we need to solve this problem. This is what you need to learn first, that kind of structure, that is what I'm looking for. And I don't think it's something that is readily available in the market or I'm not even sure most courses offer that even when they offer these courses online. So I think for me personally, I'm looking for in a mentor somebody who is not just interested in learning these skills, but tell you exactly the structure of learning. All right, this is important. For example, you need to learn A before you learn B, you need to learn B before you learn C. Something like that. That's what I'm looking for in a mentor.

I think that's the important part. So I mentor people every now and again and I know a bunch of people that want to mentor but aren't sure what the process is. And they often think it is more teaching. They think they're going to have to spend hours teaching somebody how to do something. And if you're in an organization where you're mentoring somebody in your team junior, then yes, you probably will do that. But if you're external to that organization, to that person, to me it's about connecting with somebody, listening to where they're at, potentially making a suggestion what they might want to try next and why. It's about storytelling. One of the things I find is if you can tell a story from your career that I did this and this is what happened and this is why it happened, that's valuable. And then the last one is context. When somebody says, okay, why does everybody say you shouldn't do real time streaming? The context is it's expensive still to do that versus batch. Nine times out of 10, they don't need the data straight away. They can live with 15 minutes or an hour. So everybody says they want it, but actually when they see the cost of it, they say they don't. That's the context. And you tend not to find that story of the context of why does everybody say don't do near real time unless you have to have it is not well written somewhere. And it's definitely. I don't often see it as part of a course. So, yeah, I'm with you as it really is an hour, maybe two a week, having a chat with somebody and just helping them with their career. And so for me, if I think about all the people that helped me, all the people that helped mentored me in my career, I think people who have been doing it for a while need to pay it back. So anyway, end of, end of pitch for people to mentor more people in the world. Okay, so let's talk about this idea of redesigning the current data systems to make them better for LLMs and AI. So when you kind of propose that as a subject, talk me through it. What do you mean by that?

I started thinking about the whole LLM thing some few months back. As a matter of fact, just like I told you, I have an article I want to release which I think I just wanted to talk about the traditional data systems and what we need to get to that point where enterprise can take advantage of what LLM is offering right now. And this started off from working and everybody's talking about LLM since 2023. Now everybody has been talking about LLM and the whole buzz and there's so much happening around LLM these days. So I asked myself on the job right Now, I use ChatGPT, I use Claude for a lot of things. But I asked myself, how does the enterprise, how does the business still get value from these LLMs at the end of the day? So that was what led to that thinking. And then I started exploring and I've seen very great articles and research papers around that. So for me, the management team here or where I work, they are not just interested in hype, they are interested in value. So before they put money into any of these things, they want to know how much value they can get from it. So sometimes it's difficult to get a buy in from the management about some of these things unless they see what they can get from it. So that was why I started talking about, I started thinking about how we can bring value to the enterprise through LLM. But I then discovered that with the current system, I don't think that value can be created immediately. With the current systems that we have, if at all there's going to be value. When I say value, I'm not saying we've seen Agent doing interesting things like a lot of people integrate agent to their GitHub repository. That is good for the developer. But when it comes to the business, I don't think there's so much value that they are getting from that. So I think for us to get to that point, there's going to be a need for us to rethink some of the processes and even the data system itself should. So that was how I got to that topic, to that theme that I gave you. Yeah.

Okay. And so there's a couple ways we can approach this. We can look at what you would see as a traditional data system and describe that and then figure out what you think should change. Or we could just identify some things, people, processes, technology, design that you think when you look at that, that needs to change for the use of LLMs and to add value back to stakeholders. So I'm with you. We see lots of use cases of developers using copilots and LLMs to automate or make their jobs easier, remove that drossy work. We see some interesting use cases using those tools to automate business processes data. And then there's this third kind of dimension which is agents using data for stakeholders. It's not business process automation and it's not co pilots for developers. And now everybody goes, oh, it's text to SQL, right? Ask a question, get an answer. That's the obvious use case. So which way do you want to do it? Do you want to describe a legacy system and then what you change? Or do you want to talk about specific use cases that you think are valuable to a stakeholder? And then we talk about it from that lens.

I think we can talk about a bit of the two. When I was trying to just put some points together before joining, I actually have three questions I feel like it's important for us to answer. One of the most important questions is why do we need to redesign the data system? Why is that we can't take advantage of the current system? I'm talking about the traditional systems, right? I think that is very important. And then we then need to answer the question why can't the current system be leveraged as it is? Because there are two different things. One thing is that we need to answer the question why we need to redesign the data system. We need to talk about why can't the current system be used as it is? Unless maybe we can even add on top of that, what do the traditional data system looks like? Maybe somebody's listening to this conversation and does not even know what the traditional data center looks like. So we might even want to touch upon that. And I think the final question that I think we should try to provide an answer to is how then do we redesign the system that we feel or that we think will be able to get us to that level where we can get massive gain from LLMs and agents.

Let's do that. One of the things that we need to always recognize is we need to anchor our language in a way that everybody else gets a shared language that we're using. So when you say traditional data system, I have in my head what I think a traditional data system is. And it's probably not the same as yours, because mine's based on 30 years ago, 20 years ago, 10 years ago, there's been multiple iterations of traditional data. So what I always suggest people do, right, is when they, they've got a mental picture in their head, they've got a map. It's always useful to describe the map in a way that everybody goes, oh, that's what you mean. But let's start off. What do you want to do? Do you want to do why do we need to redesign traditional data systems? Or why can't we leverage them for LLMs? Take me away.

I think we should start with the why redesign the data system? I think one thing that is really important is that we need to know that LLMs, or agent, whichever way you want to call it, they are different from humans. Right. If you have used agents or if you've used LLMs, but whether it's Chargpt or Cloud, whichever one, and then you use it around data, you will know that they don't ask once, they don't ask once, and then analyze, there might be a need for you to ask again, change some things. And so because of that, in an attempt for these agents or LLM to provide an answer, they speculate. So that means they try to do some form of exploratory work. So that makes them different from human. So because they are depending on whatever it is that you give to them for them to now think so. So that in itself tells you immediately that they behave different from humans. And so it's the natural way we get information from data system cannot be the same way that these agents get information for us. I don't know if that makes sense.

Yeah. So let me just play it back and see if I understand what you're saying. So if we look at the LLMs, the foundational models, they've been trained on large knowledge bases of text. And so the early days, we'd ask them a question, they would search that knowledge base and they'd come back with an answer. And all we did was we'd ask a question. It would look into itself like a human would, if it only could look at its memory. So it would go, what do I know about that? Here's an answer. And then what we found was actually, we want to provide context to it. We want to provide some additional information that foundational model may not have or may not have brought to the front of its memory. And so we started seeing techniques like RAG and prompt engineering ways of giving it additional instructions on how we want it to behave. Because the way my dentist behaves is different to the way a data analyst behaves. The prompts were behave like a dentist, behave like a data analyst. And then the second thing is, here's some other information you probably don't have that again, a human would ask if you said to me, how many customers have we got? I'm probably going to say to you, what's your definition of a customer? And where do we hold the data about a customer? So I can go look at it and count it for you. And you want to count today, right? Not last week or next month. And it's okay to count it to one. So 1001 customers is okay. So there's a bunch of questions as a human, you would ask if you're an expert in your domain. And so this idea of context and reinforcement is passing that back. So we pass the LLM prompts to tell it how to behave, the data we want it to look at, if it doesn't have access to it already. So we give it access, and then the context, additional information we know is valuable to it. So is that what you mean by the change in what we have to provide versus a SQL statement, which is select this from there, and all we got to do is give it the code and the data, and the machine runs that code on that data and we're done. Nothing else needs to happen. We don't need to tell it anything else, just run this code.

I think the keywords, you rightly explained it, but I think the keywords there is that because LLMs are going to be, or let me say agents. I mean, let's just use agents, all right? Because agents are going to be one of the ways we interact with data. When it comes to LLM, agents are going to be like the main use case, right? So I think the key word is that since agents are not human, there's a higher chance that in an attempt to provide an accurate response to your queries, they speculate that means they spend a lot of time doing some exploratory work, and that in itself have impact on the data system. So because there's going to be a lot of Redundant queries, things that are not necessary that you will need to fine tune and all of those things. And so because of that, I think just like you said, they have been treat on this massive test that might not even be relevant to what you're saying, but then they have to regurgitate something back for you. So I think the key word here is that they speculate and in all of these it's an attempt to just give you a response.

Okay, so again, if I clap back, if we took a human behavior, it's like me saying to you, here's my one terabyte data warehouse, go and tell me how many customers I've got. But by the way, there's no table called customer. And now you've got to do a whole lot of exploration, right? You've got to go and try and figure out what's a customer called, what table does it live in, how's it defined, and you're going to go and do all this work and that work takes human time and it takes compute from the system because I'm going to be writing queries. I think what you're saying is with the LLMs, it's the same if we say to them, here's a big blob of data with no context and no constraints and go answer this question. Because with the reasoning models and reasoning in quotes, going through and building itself a curriculum, it's building a manifest that's saying, I'm going to go try that, that didn't work. I'm going to go try that, that didn't work. And it's trying lots of things. And because it's a machine, yes, we sometimes see it tell us what we're doing, but it's doing a whole lot of work under the covers. And one of the other things about that is token costs because every time it does a task, it's using a token that costs us money. So if we change the way our data platforms are structured, we can make the LLMs more efficient, more effective and require them to do less thinking because we're giving them hints of what they should use and when. Is that the angle you're taking?

Yeah, correct. The fact that you mentioned the cost thing make me remember some of the reasons. So I wrote here, you pay for waste. And this is one of the reasons why I feel like we need to redesign. This is because enterprises, businesses don't want to pay for waste. Redesigning this system is how we convert this speculation to speed. Because it's important for us to know that businesses Whether small or large, not in any way, we allow waste. So it's important for us to know that agents, they don't just query, they probe and they continue to. And there are a lot of redundancy at the end of the day if we don't get this system right, because it is in the nature of agent to probe instead of just getting that. Yeah. So yeah, you're absolutely correct. Cost is a very important.

And one of the things is at the moment we're not paying the true cost of those tokens. We paid $20. Okay. Now we have to pay 200amonth. Okay. Now we're getting some constraints from Claude and those tools where we can't just run everything forever before we run out of our, our limit of usage, but we're still not paying the true cost. And so while we can do things that are lazy right now and they work and don't cost us a lot, it's going to change. Right. And so if you're going to start getting the $30,000 bill and we might as well start designing our systems now to be cost effective and efficient. And I think the other one is speed. If you think about back to your power the eye tableau days, there was a rule of how long you could let a user wait before that report rendered. It was a second or two and then they start going, this is too slow. And then you think about how much engineering we put in to summarize the data or to give us that performance. And now you go into an agent, an LLM and you ask it a question. And then when it's doing its reasoning models, it's really interesting to watch how it just sits there and gives you some feedback. It's working, it's not dead, but. But it's taking way longer to bring back counter customer, which I could have had on a tableau dashboard in less than a second with a blob of code. So we're still experimenting where these agents are best fitting. Okay, so what we're saying is things have to be changed because just throwing an LLM on top of your current data platform is not going to be the most cost efficient, not going to be the fastest. It is going to waste and that waste will hurt us. Okay, what's next?

I think we cannot talk about what the current data system looks like. Right. What I term traditional. And like you said, traditional in this case maybe means several things to several people. Right. But for me, traditional just mean the current system where we are able to use languages like structured query languages SQL. So right now the way we work is we leverage platforms, databases, postgres, name it, all right? And then we just run a query and then we get a result. And that's what the current system looks like. Right? But then when you look at the way agent works, I think the right word to use is that agents don't just use queries. What they do is not just querying. What they do is more like investigating, probing. They want to probe. So we need systems that are beyond. Of course, SQL is very important. Many times people say SQL is not going to be where we are, we're always going to be. I see people learning SQL every day. I still see a lot of university teaching SQL in their curriculum. So I'm not saying SQL is going anywhere, but I'm saying we need system that will accommodate for other things like natural language. Because these agents will need much more than SQL. Like we said, they would need context. We need to provide some kind of context. So there might be, maybe there's going to be a need for us to have interfaces that allow for not just the structured query language, but then maybe natural language. So the current data system that we have does not have that yet. So there might be a need for us to incorporate these kind of interfaces to accommodate for these agents to perform better.

In the past, we've always had data catalogs. So we've had this idea of a catalog that sat across the technical metadata, the technical context, the technical structure. So I have these tables, I have these columns, I have these values. And then from a governance and stewardship point of view, we always knew that adding context into that catalog had value. So that table holds a record for customer, that table holds names. Therefore it's pii. That is the table that you query if you want a single list of customer. These other ones, they don't hold everything or they've got duplicates. There's all this language, this descriptive stuff that was useful, but what we learned was nobody ever does it. The cost of creating that context was way higher than the value in using it. And yes, we had big organizations where it got mandated and they were certain industries where you had to do it. And yes, we got big tools that automated some of it and that. But in my experience, data catalogs was the tool you bought and two years later you turned off or nobody used it. I think with LLMs and agents, I think that those types of capabilities where we bring this natural language context in this text, this description of information is highly valuable. But I'm not convinced that data Catalogs of old are the right place to do it. And the reason I say that is I think they're disconnected, they're exhaust. They hoover up exhaust. So we hoover up some technical metadata and then we ask somebody as an extra task to go and add the business context, the operational context of how many rows are in there, what's the data quality like? And they don't even have this idea of agenda context. You can't really at the moment store prompts or hints for an LLM for an agent. I don't think they're fit for. I don't think they'll fit for purpose in the original days, but I don't think they're fit for purpose now. So I'm with you. I think that there will be a new paradigm coming out about how we hold this language, this description, this context against everything to do with data. And then that is highly valuable to the agents. And then the way we capture it, to me, it's got to be done as close to the creation of the code or the data as possible. So data engineers who are doing the work and have it in their brain, they know how the accounting customer. I'm going to create a dim. I'm going to create a hub and sat. I know the logic because I've asked and I've done that work. They're the ones that we have to make it really easy for them to capture that tribal knowledge into a place that the agent can go and find it. Or maybe we just have probes in our head and then the LLM can ask us. Because when you work in an organization, you probably do it. You go, oh, I don't know how we calculated that. And you go and ask Bob, and you know somebody in your team that you can go and ask and you get that tribal knowledge out of their head. Maybe we'll. I'll be facetious here, but maybe we'll have probes in our head where the LLMs ask us. Yeah. So is that what you mean, that idea of that really rich descriptive stuff? Yeah.

So like I said, in summary, I think it's important for us to take advantage of these agents. There's going to be a need for us to have a paradigm shift from the normal SQL against databases. There's going to be more things that are involved, like you mentioned, maybe catalog there, maybe need to have the data system itself. Maybe the databases might have something that stores a lot of context in memory just to provide more context. But I think one thing that is important for this agent is that they need grounding, they need to have a lot of information for them to act anyway. So the current data system we have to evolve to the point where we have some of these components integrated into the system for us to take advantage of the agent.

Let's just take that comment around probing from the way that you meant it, not me putting a probe in my head for the LLM to get my knowledge. And the traditional way we've always done it is we've taken data from source systems and we brought it into one place and we've typically put it in a database of sorts that like you said, we can write SQL and we can get an answer. And yes, we played around with NoSQL databases and Hadoop and horror of other stuff, but we keep coming back to a database columnar that stores data in a certain way and allows us to use the standard language to get the data back out. Seems to be a valuable way of working. And then we've now got this introduction of mcps where we can effectively create almost an API or a network port or an HTTP kind of endpoint for the LLMs to talk to a system and get a response. So I need this, give it back to me and it's not writing SQL. It might talk to the MCP server and the MCP server might write SQL to talk to the database and get back and then hand it over. So I look at it and I go, do you know that there's a bunch of use cases where probing won't work? And I can come to those in a minute. But why wouldn't we just fundamentally change the way we work? So rather than grabbing the data from the source systems and put it in one place, why wouldn't we just expose these MCP services and allow the LLM agents to always just do a one and done query? If you looked at it, it also actually removed the data warehouse. Now we'll come to the use cases. Why? That probably won't work at the moment, but I'm just intrigued by that. It's just go away, ask the question, give the answer. If the context is bound around it, if the context is the thing we care about, that's the pet and where the day to live really is the cattle. It's an interesting architecture change to what's been 40 years of my life, so.

I think is an interesting one actually. But of course we can talk from nanti to world. The reason why that is not going to be possible, but I think it doesn't stop the fact that it make us Start looking at things in a different way from the way we used to work. Like you said, I don't think there's anything stopping us from doing that. Even from hindsight. You start thinking about the cost effect and a lot of things that might. But I think there's absolutely nothing stopping us from exploring that data warehouse today. And this is me just digressing. In my experience, the whole idea behind data warehouse, which is consolidating data from different sources and putting it in one place. When you look at it, if we ask ourselves this question, has it really solved the problem? Because I've seen places where there's still information that you can't find in the data warehouse. Data warehouse is supposed to be the source of truth, but there's still information that you can't find in the data warehouse. So if that is the problem, so what if we have, like you described, if we have a sea of data where we can just throw these MCPs to get information, I think it would be a better. But then of course there will be a lot of things that goes into making that happen.

And a high chance of waste. Yeah, a high chance. Of course, wasted, cost, slow. All those things that you talked about at the beginning that actually we need to focus on at the moment. I think whatever we designed would give us those problems. But that's at the moment because we're in the beginning of this whole new wave and so we haven't really done the rethinking, the re engineering of what it would look like. So where'd you get to in terms of if you started with a blank piece of paper, what would you do right now to build a data platform capability that is LLM agent friendly?

The first thing is that we need a different query interface. Just like I explained, we need a different query interface and this interface should accommodate for both natural language and also the structured query language, SQL and the rest. This interface will accommodate for that. But at the same time, I think to avoid some of the things that I've talked about like redundancy, like waste and all of those things, I think there's going to be a need for us to have the data system being able to determine what query or probe needs to be executed just to prevent that waste? Because even when we have this interface that accommodates for several imputes that allow agent to run data different queries against the data system, it is important for us to also put some measures in place to know what query to execute. Because at the end of the day, whatever you get back is not going to be all useful. So I'll say an example. So if you say something like, oh, give me the sales trend of. The sales trend of maybe battery in the United States, an agent is just probably going to run a query against different web servers, different webs and just spring. Everything is not going to be useful for you. But when you take that into an enterprise environment where you need to manage costs, where you need to ensure that you maximize and optimize your queries, then you will see that is going to be a big problem. So for me, I think the first thing I will look at is that we need a different query interface and then we need to find a way that the data system determines what query to execute. So for example, the database, if there's a way to. Of course there's going to be a need to implement maybe metadata storage and all of this to give an idea of what currently exists within the data system. So that determines immediately this is what is available. So if you run a query that is not relevant to what is in the data, you just don't execute it. So that is the way it is in my head right now. So we need a new interface. We need something within the data system to determine what query needs to be executed. That is the way I will start designing.

What's interesting is this idea that when we move from humans to machine, we get infinite scale. So that example you used of what is the sales trend of batteries in the US As a human, I have some natural constraints. I have natural constraints of time. I can't spend two years going and finding that number. I have a natural constraint of knowledge, right? I'm going to Google it, I'm going to find some websites and then I'm probably going to either run out of time or get bored and I'm going to stop. Whereas when we talk about the agents, it'll scale if you let it, right? It'll search every website in the world if you let it. If you tell it to, it can run a thousand human. I don't know what is it, CPUs for the computer, miles per hour for a car? What's a agent cycle for? The number of human hours. So it can run a lot of human hours in a short amount of time for a cost. Anything we do that is a pattern based on a human constraint of I don't have time, I can't scale myself, that disappears. And we've got to be really cognizant of that or we're going to get massive waste again, like you say, I think the other thing is this idea of if you look at human behavior, if you're in a data team and you're in a team of five to nine, you're going to find that normally you'll have a data modeling expert, you'll have a data collection source system expert, you'll have a data engineering expert, you have this experts, and you're naturally going to go and ask the expert for some help. Hey, I'm looking at this and I need to model it. And I know where a data vault is our standard. So can you just give me help? Because I've never done it, I don't do it very often. Or can you peer review? And one of the things we've found as we're building this out is that when we had a single agent, so our agent is called ad, when she was the only agent we used to flutter, so we would say, go and do this work. And she's just time slicing her skills all over the place. And we always got back an okay response as soon as we then broke her out into other agents. So we have 80 the data modeler, Adi the engineer who goes and figures out the transformation rules. And then ADIE to boss. And so we talked to ADI to boss and she knows about these other versions of herself. And then she goes, oh, okay, the next thing I need to do is model that data. And she talks to AD the data model. And what that means is we provide Adi the data modeler, bounded context. We give her very specific prompts around. You are a data modeler, you're not an engineer, you're not the boss, you're not the bi. This is your job, these are your skills. We give her a bounded subset of context. You can look at the data structures, but you can't go and do data quality tests because we're talking about, in this case, conceptual modeling. That's not your job. Another AD will then take what you have done and make it better or do her task. And what we find is that specialization of skills and then that handoff which happens in a team, but you don't really see it if you then programmatically do that with the agents, we seem to get better responses. So I'm with you. I think this idea of what interface, what language, what set of skills, what Persona can be bounded and then handed off all over the place. And that's, I think, where are we going to end up? I still think SQL to query data, if we need to do that is going to be the answer. However, We've had great success giving images to the LLMs, not data. But it's expensive, it's wasteful. So if I give it a photo of an image screen, capture of an Excel spreadsheet, it's going to do really well with it, but it costs me a lot more tokens than if I give it the csp. So I think it's again that trade off between art of the possible right now and then the cost and waste of doing cool stuff.

So I think part of the reason why I also mentioned that it's important for us to have this interface that accommodates for both natural language and also SQL is right now part of the way I work is I actually have bunch of queries that I've written in the past and I just dump them in my LLM, dump a lot of them. And right now I, I just ask questions and because this ll already have history of my queries that I've used tpv, how to calculate tpv, how to calculate TPC and all of those things. So I just say, hey, can you give me develop a query to. And it does that and then give me the output. So I think part of what you're saying is providing context is there's still going to be a lot of work to be done around contest. So giving queries that we've used in the past, or maybe anything that can provide more grounding, we actually help resolve some of these issues. I've seen it work firsthand, it's working perfectly for me. There are a lot of hard requests that I don't spend my time developing the query anymore. My LLM just does that for me.

And that's a really good point. It's this multimodal, I think Joe Reese talks about multimodal arts. And so this idea that no organization actually uses one data modeling technique for, for their warehouse. They may say they're dimensional star schemas, but there's some relational stuff in there. There's some sorts. Yeah, we always have more than one modeling technique and I think we're going to end up with more than one language and that multimodal language is going to be important. And so I'll give you an example. We presented all our documentation to AD for our product and so we describe what we call change rules, which are transformation code. And we have a set of language you have to use for it. And so we described how the rules work, what the structure is, what the structure of the language is. And there's some examples if you want to Go and pivot, do this if you want to unpivot, right? But it was all in text. And so when you know, somebody comes in to our platform and they go, I need to transform this data and it's right in plain text, I need to transfer this data. How would I do it? She came back with an okay answer, but it was just okay and sometimes it was just wrong. She would hallucinate and we're like yeah, that's not how our product works. And then what we found was effectively we run a multi tenancy architecture. So we stood up another tenancy or one we already had for our partners called alliance and we pushed example transformation code, our example change rules across every customer we had. We took the rule itself, not their data and we said this is a rule we've applied before. So your exact same example as yours of here's a blob of code, this is a rule that we've used before. And we then created an MCP server that AD could see to see those rules. And that's when we broke off to AD the change rule. And so all she does is if you ask her how you can transform this data in natural language, she will go and search those rules, but she's effectively searching a code repository that is highly opinionated and highly structured. The language is the same, what's in the language is different and the response back we get is so much better now. And so the thing we found though was while we had the rule, the logic, the language of the rule, we had no context, we never wrote down why we were doing that. So as soon as we added that in natural language of this rule is to take stats, New Zealand data which is being summarized as columns and unpivot it to rows so that we can use it later when we need a row per record, then she's ah, now she's getting both. The thing, the same as you're saying is that if you take your blobs of code that you use on a regular basis and I think you, when you talk about TPV and tpc, you're saying there's some metrics in there. If you then in that code put in the definition of the metric TBV equals and then a bunch of text and that's as a comment in that code, then the LLMs are going to get both a structured piece of code and context. They're going to get SQL and they're going to get natural language. And I think that's where we'll end.

Up going back to your question if you ask me what are the things I need to think about if I'm going to design what this system should be, I think the interface is going to be very important and also how to manage what query needs to be executed to avoid waste redundancy. And I think that is the way I'm going to think about it.

So let's take the example where you've got you and blobs of code that you have found highly valuable and you've already tested putting that code as a reinforcement model, giving that context of the code to an LLM, have been able to ask questions and get back the help that you needed. And let's say that you do extend it out where you actually define what those metrics are in natural language so it has a richer context. And then you've got three other people in your team who need to do the same thing. Like right now, what would you do? Where would you store it, where would you surface it? What interface would you use to create it, to share it? Like, how would that work for you right now?

Now the way it works, this is just for my personal use. I've not had any reason to share with anybody. But I think that also points to what I was talking about when it comes to conceptualizing what this data system should look like. I think there's going to be a need for us to have part of the data system, maybe the database, a part of it that stores these. I don't know whether it's going to be a meta store or whatever that stores this information. And the reason being that regardless of who is running this query or who is submitting a probe, they have access to the same information. And that information can help provide more grounding for the LLM or the agent or MCP or whichever one that you're thinking. I think that part of what we need to think about is a part of the data system. Maybe this can be the database now, maybe this open a new opportunity for database research to see how we can a part of it can store information that provides more ground into dlm. So that is the way I'm going to think about it. But right now I've not had any reason to actually explore.

It's an interesting one because if we think about the fact that we've always stored our operational data, OLTP data, separately from our analytical data, our OLAP data, that's because in the past we never really had a technology that allowed both to be stored efficiently in the same place and queried. They were two different query Patterns, two different storage. Storage patterns. And that's not true anymore. We've got tools out there like single store that say I don't work with them. So I don't know if it's true, but they say they do that, yet nobody's really adopting it that I can see. We still keep it separate. And so when we start bringing this idea of a context store, vector databases seem to be the thing. But actually, a lot of the times you don't have to use those. You can use anything. So I'm intrigued, like you, to see whether we end up with yet another database or we now end up with databases that store the data and the context side by side. It's an intriguing place to be. And then do we end up with a thing that I call the context plane, which is the idea of a shared, centralized layer of context, or do we end up with a context grid, the idea that context is stored next to the data and then something else federates. It provides a grid type architecture. And so again, we're at the early days where lots of people are exploring and saying what works and what doesn't work. So just on that, just to close it out. So this is something you've been thinking about in your spare time, right? You're not part of a software company building this, your organization. You're not part of a team building it out for them. This is something that just natural, inquisitive nature going. Yeah, that's cool.

Yeah, pure natural, inquisitive. I've read a lot of papers. There's this interesting journal that came out of Berkeley. I can't remember what the title is now, but it's very interesting. They did a very fantastic job around this conversation, too. So, yeah, for me, I'm not part of any team actively developing something in that space, but it's just something that I'm just interested in.

And the next step is you're going to start writing. You're going to start writing your thoughts, starting. Right. Your exploration.

Yeah, of course. That's part of what I'm doing. By the time you release this, I want to listen to this conversation again, because some of the things that you said, I think they're really interesting. I want to look into them more deeply. So, yeah, I'm just documenting my thoughts right now.

And my advice to people is document lightly and document early. So share lightly, share early. Don't wait for this podcast to come out. Document what you think right now, and then write it up, share it, and then think about it a bit more and then write it up and share it. Because people often find the journey of the way you think as interesting as the answer. And what I say is, by writing, it's forcing you to think with clarity. You think you know something, you're like, oh, yeah, I think I know how that's going to work. And then you write it down and you're like, yeah, no, that's bollocks. That's not going to work. So that art of writing just makes you think. Writing is structured. It has beginning, middle and end. It's going to force you to storytell to yourself and validate what you think is going to happen. So, yeah, my recommendation to everybody is write small bits, push it out early. It helps you think it'll change, and that's okay. It's not a book. It's not something that you can't change. It's, hey, I thought this and now I think this. And I think that's better than what I thought before, but I had to jump from A to B to C to D to get to E. All right, when this does come out, how do people find what you're writing? Have you worked out where you're going to publish it?

If I'm going to post, it's going to be on my LinkedIn. If I'm going to leverage any other platform, I also put it on my LinkedIn. So by the time I'm done with this, I think everybody can find it on my LinkedIn.

Excellent. Most of us are now writing on substack because LinkedIn sucks for long form content. So I will encourage you to create a substack. Yeah. And then tell everybody on LinkedIn that where the long form content is, because LinkedIn's kind of killing the ability, which is really sad because actually I'd rather just ride in one place. Excellent. People can see how you're exploring and what you're learning and what you're sharing. I look forward to it.

Yeah. Thank you.

I hope everybody has a simply magical day. And that day didn't just was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to agiledata IO.