Listen now | Recording of AURA - Chris Gambill & Shane Gibson 14th August 2026

Hey, Chris, welcome to another Aura Ask Us Anything show. And sorry for everybody for being late. We had Substack technical difficulties, but we got there in the end. How are you going? You've been back-to-backing today, haven't you?

I have been doing all the things today for sure. I just got off with Scott Haynes from Databricks doing— we created like a synthesizer with Omnigent. It was super cool. Like just vibe coded a Yeah, like a synthesizer, like a, just a software synthesizer. It was super cool.

Oh, wow.

In 40 minutes from scratch, prompt it and it created it from scratch and it worked.

That's one of those weird use cases for using a data platform for.

Yeah.

I won't ask how many butterflies were killed while you did that, but that's okay. Excellent. Hey, I think 2 subjects for today, unless people pop on and give us something else in the chat. So I think we got some questions. Hey, Matti, feel free to ask questions if you want, and we'll talk about anything you want us to. But at the moment, we had some comments earlier about the difference between context planes from a technology point of view. And, you know, that kind of thing, and then how to collect and gather context that would go into it. So, that's one. And then, I think you and I have been kind of talking or working on master data management, golden record stuff. So, we might as well talk about that as well. So, which puppy do you want first?

That's just coincidental. We didn't even orchestrate that. Like, that was just happened to both be working on it. I mean, we could jump into the MDM piece first and then try to go from there.

Bringing back memories, right? I mean, it's been a while since I've done hardcore MDM. Like, I thought it might have been dead, right? Master data management.

And that's funny. Like, I saw that the other day. And from my perspective, like, I think we didn't call it master data management for a while. But like, the concept was always there, right? People struggled with it. We just called it like unified customer. We called it Customer 360, or we called it something else, but it's really all MDM, right? In fact, even to that point, like, when I started my coaching program, like, I have my people do 3 projects, right? 3 custom portfolio projects. And the last one, I always make them do like MDM with, deterministic matching and probabilistic matching and understanding what the differences are and benefits and costs of each one. And, like, it's such an important topic for businesses, I think.

Yeah, I remember I used to work for SAS, the software vendor, many, many decades ago, before I had gray hair. And they had a subsidiary, DataFlux, that they bought, which was quite a cool tool. I really, really liked it. And it stayed a separate company from SAS, which is probably why I liked it. And what was interesting was that was used a lot for data quality, but also master data. And so, back then, you know, we're trying to— I can't remember whether Dataflux was probabilistic or deterministic, but it was one of them. And so, we had to explain the 2 differences, and then show them how it worked. And it had the whole, you know, dictionary, all that kind of stuff. But yeah, I don't know whether master data management's as a term is so common now. I mean, when I talk to people about it, they say they want single view, right?

Yeah.

Golden record. And the first thing that I say to them is, okay, these 2 flavors, 2 patterns, which would you like? And the answer typically is both. But I ask them if they want a single list. So, if we have customer in, you know, CRM and in service and in finance, so 3 sources that hold customer records, not keyed the same across the organization, because, hell, they never will be. And we want to bring those back, and we want to say they're the same and have a single list of customers, right? That's a whole bunch of patterns we have to do. And then the second one is single view, which for me is saying, I've got a customer, and they've got transactions in lots of systems. So they've got transactions in CRM, and they've got transactions in service and in finance. And I want to bring all those transactions, all those events, all that data into one place and be able to see it against that one record. And so, I don't know about you, but I often find we have to do both, right? We have to go and take a bunch of concept records and match them, and then we have to take a bunch of transactional or event records and then combine them and store them against those concepts. Is that how you see it?

Yeah. Well, and that kind of goes to the matching piece of MDM and then the survivorship piece. And then you're layering in the transactional piece, right? So, which is how we get to facts and dimensions, right? I mean, that's—

Or how we get to hubs and links, yeah.

Hubs and links. We'll call them hubs and links, because it definitely gets more complex with MDM for sure. But yeah, like, when I've worked with some, like, Fortune 500 companies, like, to your point, they have HubSpot, they have Salesforce, they have the random marketing team that's going out to events and keeping things on paper and bringing them back and putting them in Excel and wanting them uploaded. And then they always have some ERP system, right? They have their, you know, their finance piece. So like a NetSuite or something. And every one of those, like one company I worked with, every one of those had a different version of Tech Mahindra. Mahindra, Tech Mahindra is a company that operates like globally. And Some in some systems it was Tech Mahindra Limited, and in some systems it was just Tech Mahindra, and in some systems it was like somebody abbreviated Mahindra to just Mahind and like cut off the last part of it. And you get that right? You get that whether that is because there's multiple different systems or you get that you know the article that I put out this week was you know. You know, if somebody puts something in multiple times in a single system and maybe they misspell it, you know, 10 times and 3 times it was correct, which one of those records is the one that you want to survive? But this one company, the Fortune 500 company that I was talking about, like they needed to be able to see, okay, this is the cost of acquisition, this is what we're actually making from that customer, these are residuals, this is— these are where it ties back into different opportunities. at a global level for that particular customer instead of at each of the individual. Think of, like, Walmart, right? Walmart has a ton of different stores, and then they have a master account. Anyways, I talked a bunch.

Yeah. No, I mean, I think we'll come back to that hierarchy of matching, and then also the idea of what is a master, and then the survivorship rules, right? But we've got a question from Maddie. So, what kind of reference tables would you implement in every single company you're working for? I'm interviewing for DEs now, and they are surprised when I even mention one calendar table for the whole team. Ooh, here we go. So that's, um, calendar— when you talk about a calendar table, I automatically think, uh, dimensional modeling, star schema.

Okay.

Because from, you know, we have a calendar dim, right? And we need to be able to create that. Uh, funny enough, if you use Data Vault, we don't tend to hold, uh, calendars or dates as a concept. We hold them as attributes, as detail. Um, so it's really interesting, like, yeah, as soon as I hear calendar, toggle and I hear dim, I'm like, yep, that's a lookup. Okay, but you go first. What do you reckon? Like, what types of reference tables? You can answer that many different ways, but yeah, let's have a go and then we'll get Mehdi to come back and see if that was actually the question he asked us.

Yeah, no, so one calendar table for the whole team, for the whole, for a whole org, right, is, can be okay. I think it depends on the org. So I've had situations where I've had like a— we'll stick with the kind of a date dimension type situation where, to your point, like if you're in Data Vault, like each domain, each kind of vault is going to have its own attribute for what that date looks like, right? Because you have these concepts of calendar year or fiscal year. Right? Or which throws everything else off. And being able to track things by quarter, by week, by month, by day, and how granular do you need to get there? I went way down a different rabbit hole for a second, so I need to come back out. But I did. So kinds of reference.

Well, hold on. So let's just take that one. So In the past, I would, when I was dimensionally modeling, right, I would hold a reference table, which had my calendars, and I'd say, I'd tag each of the dates, you know, with those date parts, fiscal, calendar, calendar, you know, calendar, calendar, public holidays, day of the week, right? But what we actually do now in BigQuery is we just treat that as calculated attributes, because I can, Well, for the months, weeks, years, and day of the week and that, because I can basically create a calculated column that just looks at the date and gives me back those values whenever I need it. So we don't do the lookups from there. Now, if we were doing event dates, yeah, so specific, you know, retail holidays and those kinds of things where the BigQuery functions by default don't do it, we may use a reference table and do a matching. But we may actually create that as a function that we can call that then calculates it, right? So, they're a little bit more kind of different when we kind of look at that one. So, yeah, so I suppose the key is, if you're interviewing somebody and they say, hey, they have an alternative way of dealing with dates, then that's okay. If they're like, what's a reference table? Then—

Then that's a problem.

Well, yeah, as long as you're hiring a junior, and you've got good training, then you're okay. Right, you carry on where you were going with that, 'cause I've got some other things around reference tables I wanna jump in afterwards.

And to be fair, I'm— Matty's— I talk to Matty every week, and so I know that he's hiring for seniors. So yeah, that's my— I have a little bit more context there, but yeah, that was, I would agree. Oh, I remember where my train was going. Like right now from a Databricks perspective, we want, you know, when you were talking your favorite topic, medallion architecture.

Nope, it's not an architecture, it's a classification system.

You know, in the gold layer, a lot of times we're doing star schema type stuff, right? And so we have a calendar dimension And that has all of those different columns, a single calendar dimension that we're using across the org, right? And what we do is often tag each of those columns with a specific tag so that somebody from sales doesn't necessarily need or want all the noise from all the financial columns that are needed for that calendar. dimension. That way we could govern it. However, with metric views coming out, in my head, what I've started to consider and think about is, do we still need that? Or can that be a metric view that we're setting up and then similar to, you know, functions or something, it's actually calculating it on the fly when the report's being run?

Yeah, and then you've got a couple of challenges there as well, right? So, the first thing is cost. Is it cheaper to join and store or query across joins than it is to calculate on the fly? So, that's one thing I'd look at. Change. So, if those reference tables are temporal, if you actually have to be able to change the data itself, and you need to be able to track those changes, you pretty much don't wanna do that as a function, because now the function's gonna have a massive amount of statements in it, right? And so you wanna bring that back to data. So one thing we tend to do is we bring the reference tables into our history layer, into our immutable persistent storage, and we treat them like any other data. So we actually effective date each record, we then track changes over it, because that pattern just makes it easy for us going forward. We tend to apply those reference tables in our design layer if we can. But sometimes we will actually only combine them in the consume layer, depending on how late we want to bind those changes, like how often, and will people restate. So sometimes, you know, we have ones where we have a bunch of codes, and there's a bunch of descriptions, and we've got one of those at the moment. And we know that those descriptions will change as people improve the quality of them. And so we'll late bind those reference tables, we'll bring them right to the end of the consume, join them there, and therefore one change restates over all time, and that's okay. There's some other ones where we go, no, actually, it's the point in time that that reference data was created or effective for, that's really important. So then we'll, in our designed layer, we'll put it in there because now we're actually matching up the history stuff. And then the question, again, architecturally, you've got to place these principles down as principles or patterns that people can use. Because I often use it, like, for MDM, I often use it for matching, you know, this equals that, right? So I don't have to write a big frigging case statement somewhere, right? I'm like, no, no, just give a reference table where I can maintain those relationships because they don't exist anywhere else, right? That's the key, is that data. We often use Google Sheets as a way for people to maintain their reference data. Because we can expose that to BigQuery without having to bring the data in, if that makes sense. So they kind of become external tables. So, yeah, when you talk about reference data, right, you start going down massive rabbit holes, especially architecturally and platform-wise.

And you hit on a key point there too, I think. Like, I'm a big fan of table-driven architecture, right, where you're storing things in a table as opposed to storing them in code. It's just architecturally, I feel like it's more business-friendly too, right? It's just, especially when you're even outside of calendars and dates, like KPIs and metrics, right? Calculations for KPIs and metrics are always in flux. Those are things that are living, and you need to be able to track them, to your point, over time. of, okay, this metric was this between this date and this date, or this KPI was this between this date and this date. And yeah, I've even built tools where we layer in an interface for the business so that they could add and adjust KPIs as they need them. That way it's not a full software development lifecycle. That way the data team doesn't get blamed for being the bottleneck in the process of creating a KPI that the business wants to track.

Yeah. It's interesting, as I'm— so we're doing some work on the metric stuff at the moment. So today, I was just doing a mixed biking experiment with Google BigQuery Graph and their, what they call metrics, I call them measures because it can only do sums and averages. It can't actually do any complex calculations yet, fingers crossed it will. And it kind of reminded me that, Every time I'm using Claude to help me do some work, it is constantly writing the metric in SQL itself. Yeah. And I'm just, I'm creating this reinforcement process for it to say, no, the metric is stored in context. You need to go and see it and read it. And then I don't care if you write the SQL to execute it, but actually, you can't make the thing up, right? Like, don't keep recreating it. And it's When it does it, it kind of makes it the same, but it's not looking at the master of my metrics, and I'm like, they're over there, right? I don't care who executes it, because that's cattle, but I certainly give a shit about who masters it. Right?

Yeah, 100%. And to the point of, like, the history tracking, having that when we do implement, like, that front end for the business, tracking who changed it. And making sure that it's governed as far as who's allowed to change it. But yeah, because we care about who changes those things, right? And because sure as heck, if something goes wrong, right, the VP or whatever that's looking at that report's not going to their sales manager or whatever that changed the KPI. They're coming to the data team to ask questions about, hey, why is this? Why did this change, or why is this wrong?

And if we think about that, I mean, because I do laugh when people say that AI will replace data engineers straight away. And then we see those, you know, those sexy demos where, you know, they ask a natural language question of how many customers they've got, and it works. And then you sit there and you go, so, hold on, let's just put all these Lego blocks we just talked about together. So, I have customer in 3 systems, and I need to bring it in and give me a single list of customers by doing some matching to say those customer records are the same. And then I'm going to get transactions. collections from all those other systems and then combine it so I have a single view of everything that we did with that customer. Some of that data might be driven by reference tables, right? So we have this set of manually maintained tables outside those systems that are saying, this is this, or this is these dates, or the series of flags, and they change over time. So we know that, you know, a month ago, this was the rule, and this month, this is the rule. We change our ETL code, over time, right? So we have these version sets of ETL. Our data changes over time. So the data we're collecting in from all those source systems is temporal, and it has effective dates and change records. Our business logic that we use in our ETL may change. And then our metric definitions will change. Now, if you think about all those Lego blocks, all those potential changes to one record that can affect the number the VP gets, That's highly complex, right? And so, it's kind of interesting to see whether the GenAI tools will be able to deal with that complexity better than us. Yep.

Well, yeah, I sat through a demo yesterday of metric views and Omnigent, not Omnigent, of ontology, Databricks ontology, and just layering the two together. And to your point of things being just SQL. The metric views there are just SQL. If you could do the calculation in SQL, then you could create that metric for it, which will be cool. Having enough context to put into your ontology layer and for it to be groomed in a way, that when that COO or CDO or, you know, your CSO or whoever goes out and asks a plaintext question, that they're going to get a correct, consistent answer for what they're asking. That all plays a role, right? Having those different layers within your system that work together. Like, that's gonna be huge in the future, but none of that replaces data engineering, right? None of that. It evolves data engineering, but none of it replaces data engineering.

Yeah. I mean, I think it should accelerate us or assist us. You know, we'll see what happens when we get to automated AI and whether that works. But again, if we talk about that context, right, where do we find it? Yeah. So, one of the things that we've been quite lucky is because I've been building our way of working and teaching and coaching it as well, I have a lot of material that I've used to coach teams to do things the way we do them, technically, or in terms of team design and our value stream, the way we kind of daisy-chain the work in our factory. And so what I found is when I'm building out the context of how we work, I have this content that I can feed it. But I've still got to then go and actually refine it.

Yeah.

So one of the things is Nigel did a little bit of the context guides and somehow it brought up the SED-2 word. then it just started seeding it everywhere. And then it kept going, I can't find any facts. And I'm like, because we don't run a dimensional model, you dumbass. And then it took me a while to go, ah, that's where you're getting it from, right? So I had to go back and edit the guides, effectively the context, to take those words out because it's a language model. And, you know, but like your survivor record, it saw that word too many times and it's like, that's what it goes. So, If I think about organizations I've worked with, I don't see a lot of documentation about the way they work, right? The policies, the patterns they use, the principles they follow. So, I think, you know, to answer that question we had originally, it was like, how do we get context? I think we need context about the organization, and I think we need context about our way of working, and we should treat those as 2 different things. And if you don't have a way of working context, you should start crafting it. Because I think that is gonna help the agents work the way you do and understand what's been built before.

Right. But way better than any single engineer could do it, right? It's a— this is a group activity. This is not a data team activity. This is a full business activity to get that context. Yeah.

And there's a really good technique I used to use with teams when they were hiring, not so much now, so maybe it's with agents, but What we used to say is, do a draft of a piece of documentation. And then when somebody joins the team, get them to read it and follow it. And then they have to update it, right? They have to find something in there that is missing or confusing, and they have to iterate it at the time that they're effectively reading it, right? So, at that point in time, not a month later. And that constant iteration meant we got really, really good documentation. So maybe, maybe for, you know, nobody's hiring these days, right? So maybe with agents, let's write it, run it against an agent, see what it does, get feedback, update it, run it against— it's almost like a TDD, right? Yeah. Yeah. Yeah.

Yeah. I like that.

Yeah. So, right, let's jump onto context technology and then maybe we'll come back to MDM. So what do you think, right? Like everybody's talking semantics, context. Obviously, you're talking about OmniAgent and ontology. and all that, but are you really— and harnesses— are you really seeing anything nailing it at the moment? Because I'm not.

Nothing, nothing that really nails it, right? There's just like, obviously, I'm a Databricks person. We've had that discussion. And I think that the Genie ontology will be helpful. But yeah, nothing that is like, oh yeah, this is gonna be the thing, right? It's not. I like the concepts, I like where everything's going, but there's still work to do.

So talk me through that Gene Ontology, right? Because I could read it as a platform that has a structure, a schema, a product. Effectively, it allows me to capture an ontology, right? Capture words or terms, definitions, relationships, and that kind of stuff. But I could also read into it that there's an out-of-the-box thing where I can ask for an ontology, and it's gonna give me the content itself, not just the tooling to create and store the content. What's it doing?

So, it does. So, I was lucky enough to see a demo of it. Basically, there's a couple of pieces to it. Just like you said, there is the knowledge graph type ontology that happens where based on who is asking the question, because it's layered in Databricks with Unity Catalog and all the governance that comes with that, Genie will understand, okay, this is somebody in the sales org. This is the— hopefully you have everything tagged, right, that you've uploaded into your ontology layer in Databricks. Databricks is also coming out with this new concept called Pages, which is kind of like the Confluence Pages, right? And you could upload Markdown files, you could upload PDFs, plain text files so that it has all this context, right? And there is a concept of a trust value, right, based on who has contributed that document, who the owner of that, you know, creator of that document is. If they have other things, we'll stick with sales, right? If they have other things in the sales org that they contribute to, Then their trust ranking would be higher than somebody that has one thing sitting out there. And you can thumbs up or thumbs down when you get an answer, and it will raise or lower that that document's trust ranking so that the next time that Genie comes through and starts to look at that ontology layer, it will. maybe pick a different document because you thumbs-downed that document the last time. So it's kind of that learning loop based on that feedback that you provided.

So again, the thumbs-up, thumbs-down come out of our catalog friends, right? That was a feature out of catalog technologies. And then I just wonder about survivorship. So if I'm vibe coding with Claude to create these markdown files and then popping them in there, and then I'm getting volume, right? I've done lots of sales documents, even though they were all completely vibe-coded. I'm gonna get up the ranking, right? I'm just kind of thinking that through. And then I'm not— so this is interesting. So when you say ontology, but then you talk about storing documents, it kind of looks like they've got a platform for context, but they're calling the platform ontology, right? Because—

It's an ontology layer that looks at that context, right?

Ah, okay. Right. Okay.

Because I think if, you know, if people use the context to graph out the ontology, right? Okay.

All right. Okay. I'll be keen to see how it does that. Yeah. I'm still of the opinion that actually human in the loop, I think that type of context is actually the thing that'll make or break the GenAI systems. And therefore I think it needs to be a human in the loop. And that, in theory, that's what the Anthropic paper that came out said as well, right? As they tried to generate it and, you know, got sick, was it 65% hit in the evals? And then they put a human in the loop to write it properly and they went straight up. So I'm gonna be intrigued by that.

But 85 or 87%. Yeah.

I mean, it was— Yeah. So yeah, something like that.

Yeah. Something in the 80s, I feel like.

Yeah. Yeah. Yeah. Okay. All right. Well, I'm with you. I think, I think we're still in the early days of context technology. You know, we're seeing the catalog vendors try to move into it. We're seeing the metric BI semantic layer vendors try to move into it. We're seeing all the platforms move into it because they know how important it is. Yeah. So it's kind of a free-for-all.

Even Jira, right, with Rovo. Rovo will read the Confluence pages to try and determine what you're doing, right?

And hopefully it reads the last updated date, right? Because the number—

So that could even bring us back to the MDM strategy piece, right? And when you're talking about deterministic, you know, rules and stuff, what do you— how— what deterministic rules do you use?

Master context, right? There we go. Master context. That's a whole new category that we've just invented.

What survives, right? It's, yeah, your MDM strategy of what survives in your context layer.

So actually, it's like patterns, isn't it? Really.

Same pattern, right?

It's the same thing.

Yeah, the— all of these things kind of grouped together, and you have some type of trust hierarchy of— in this case, it's a first contributor, but, you know, historically it's been whatever system it came from— of this attribute is mostly— this trust has a higher trust value from Salesforce, and this attribute has a higher trust value from your ERP system because it's the billing address. And well, we're getting money, so they're being billed and they're receiving their bill, right? That was probably more true back in the day. Now people get e-bills and probably goes back to the email address.

But address, email, and phone number is still a bit of a bitch, right? Especially in fraud use cases where they don't want you to find them. So actually, That's the third master data pattern, though, right? So, I talked about single list of a concept, so, single list of customer, supplier, product, single view, so, you know, all the transactions for that supplier, all the transactions for that product. But the third one is surviving attribute. So, I now have a list of customers, and they are a single list, but that customer Bob, has a phone number in 3 systems. So which phone number is the surviving phone number when we do that, right? And again, it is the same techniques, but it's another use case or pattern that we have to apply that hierarchy of and survival record patterns to it. Yeah.

And that's, like, that's such an important piece in that whole MDM concept, right, is what survives and how do you determine what survives? Right. And that's— I go back to that, that is a data engineering function and it is a business conversation, right? It is sometimes what survives needs to be a conversation that you have with like a COE, right? You know, kind of your Justice League, for lack of a better term, right? Because I'm a big nerd. And So, you put together your Justice League and you've got somebody from marketing, somebody from sales, somebody from finance, somebody from each kind of business unit that you could kind of hash out for each attribute, what is the source of truth? I hate that statement, but what is your source of truth for each of those attributes? And then the data engineer takes all that context back and sets those deterministic rules based on that information.

Yeah. And so, again, I tend to work on deterministic rules first if I can. So, code that says, if I see this, it's that. So, one we're doing at the moment is content titles for a customer. So, they fund media content, And so, when we're grabbing the data from, say, linear TV or video on demand, we'll have Shortland Street, and we'll have Shortland Street bracket R close bracket, because that's a rerun. Now, actually, funnily enough, they want to see both, but that's always good. But we know deterministically that we can then encode something that says, if we see some words up front, and we see R bracket R bracket, we can infer that And take that off and match it, right? The same with episodes and a bunch of other things, right? We can say if we strip out that episode pattern, we know that that's the same content title. But then we get to YouTube. And the YouTube content, the publishers change the title on a regular basis because they want to get more reach and engagement. And so that's a lot fuzzier, right? It's not a rule. And so the kind of, that's another level of matching. And so we can't deterministically do it first, right? But—

So for YouTube, you should, so that, and this is getting into the weeds for a second, but for YouTube, there should be an underlying ID for each of those videos that doesn't change.

Yes, there is.

And you should be able to use that, right, as your matching criteria. But to your point, if you're using titles, yeah, there are some platforms that people are changing their titles all the time to try and either— and even in some, right, if you're doing like A/B testing in YouTube, it might show up one way to one person, one way to another person, literally in the same second, right? Have a different thumbnail, have a different title. In the same moment. Yeah.

And so, one of the things we're kind of experimenting now is rather than spend a lot of design time upfront, trying to figure out what the deterministic rules are. So, we have a bunch of simple ones that, you know, we'll take, as always, the 80/20 rule, right? We'll take care of 80% of it. We've just been prototyping this idea, and I'm sure I kind of, was that Dataflux? Maybe, maybe not. Maybe it was zero, I can't remember. But this idea of treating it as a reconciliation problem, right? A left-right. So, giving a person in the loop saying, here's what the master is, and here's what we think the match is, and here's the way why we think it's a match, accept it or not. And then using that data, their click data, effectively to help reinforce the model, the fuzzy model, right?

The fuzzy model.

Yeah.

Which brings us to the 4th pattern for MDM, right? So you do your best to match, you do your best for the survivorship, but inevitably things fall out, right? You cannot come up with deterministic rules and fuzzy matching is not going to catch everything. And so you put into place a pattern, that feedback loop pattern, right? you expose and say, okay, business stakeholders, here's what fell out, what matches, and let them determine. Okay, then give them some button clicks to be able to do what they need to do there. Or like, and there's kind of 2 patterns here, or you reinforce like data grooming, right? And you have them fix that in the source system so that the next time that MDM pipeline runs, then things match the way that they're expected to.

Which is the way we should do it, right? Because then any— yeah, because then the system's right. Anybody that uses that system sees the right name. That's really where we should be.

Yeah. And I would— so I'm a big fan of, okay, this is how it should be done. This is how Works in reality, right? We've had these discussions before too. But, but yes, I'm a big fan of pushing it back to the system, getting it right in the system. And sometimes the political environment in a business, sometimes the time constraints or whatever excuse of the moment is for the business, whatever, you know, challenges with the businesses, sometimes that button click of, okay, yes, these match, is the only feedback that you're going to get, and you're not going to get them to actually go back to the source system. Depends on the support that you have with the other leaders of the business, right?

But actually, I'm wondering, I'm wondering if things like MCPs now, whether we could actually close that loop now.

We could.

I've taken that action. Yeah.

Built systems where we give them the button clicks and we say, okay, this is what matches. And then we enrich that data back to Salesforce, right? And we're, as a data engineering function, actually pushing back into those systems, those corrections that have been determined by the business.

Yeah, rather than submitting it to a data stewards committee that never meets once, meets once a month, so, you know, and always too busy. Yeah, I wonder if that's the answer now is like just, and with everybody wanting to use the MCPs to read the data, right? The fact that we can use it to push back is actually quite valuable.

And that depends on like the size of the business, right? A smaller business, probably more cost efficient to just get them to push it, to correct it in the front end. And do you, where do you want to incur that cost, right? You're going to incur that cost somewhere, either in man-hours fixing the front end or in compute programmatically doing it. And it depends on kind of the quality of your data too. If your data is crap, right? I mean, and you're spending— and there's not that reinforcement coaching loop back to the business of Let's make sure that we're trying to do everything, our due diligence upfront of putting stuff in. Then I've seen where businesses don't have, where data teams don't have that support to push that coaching back to the front end people doing the data entry, then the compute cost continues to scale and increase as well to make those corrections because they just The business learns that I could just put in whatever, and the data team's gonna fix it on the backend. Data team's free. So, it's one of those things that you have to balance and make sure that there is some type of coaching and feedback loop in the business that's supported by your leadership.

And then one of the other patterns that we used to do was the idea of dictionaries, right? So, you know, William, Bill, same term. So, we used to run these dictionaries of common terms. Now, I mean, I'm just guessing we can just LLM that, right? Like, we don't— we're treating it as almost an on-demand function. We don't actually need to hold these dictionaries of terms and their relationships anymore because we just call an LLM and it— would it be probabilistic? Well, yeah, that's the question.

It'd be probabilistic, yeah. Yeah.

Or do we keep it deterministic? Do we keep the dictionaries and use the LLMs to help us draft them?

So, so funny story. So, so many tools, even when you're using a lot of those tools out, those kind of data dictionary type tools out there, whether it's Databricks or Fabric or what's that other big one? I don't remember. It'll come to me here in an hour when we're done. But literally there is a button that you click that it auto-populates with AI your definition. You have the chance of deterministically filling it in, Or probabilistically hitting that button and letting AI assume what's going on there. And I would— I've seen people that just go through and just click that, the little magic wand, and hit save and move on to the next thing that they're curating, right?

Yeah.

Anyways, so—

And imagine, again, we're back to the model, the context model, choosing the most active person in that domain. And now Mr. Clicky, clicky, clicky, clicky is winning because they're seen as trusted.

Yeah. Yep, exactly. Because, yeah, because it looks good. Somebody else looks at it and they're like, yeah, that looks good. And then they've all of a sudden done 50 million of them.

It's a bit like how many tokens burned and how many PRs you've committed, right? How many code commits? That's a sign of Yeah, if you're a top contributor of something, quality or not. Yeah.

Yeah.

Yes.

Now I'm trying to look up the name of that data dictionary company.

2 AM. It's 2 AM, you're gonna be like, bing!

That's the magic time, right? Oh my gosh, I can't believe I can't remember it.

That's right. If you stop thinking about it, it will come back to you.

I'll stop thinking. Yeah, we'll come back to it.

And then, so we've got these different rule sets, basically, right? Some of them deterministic, highly trusted, we put them up front, then we get down to some more fuzzy, lots of other fuzzy things, distancing algorithms, a whole lot of other ones. And so again, we've got to be able to observe which of those rule sets was applied to match that record, right? So we should be logging that, right? We should be saying, when we did this match, we used this rule set, and therefore this was the thing that took the action to match it. And potentially, if there's a human in the loop, this person said yes as well. Because otherwise, we've got no idea what happened, right?

Right. Well, and I think that that's a feedback loop too, right? If you have your system, if you do, you know, we talk about build versus buy, and you build it, you should have part of that build be that one, like you said, we're stamping that rule that was used to do that deterministic match. And then over time, you're curating that too, right? Of, okay, we had this deterministic match, but the business pushed back on 10% of this. Do we need to revisit that deterministic match, right? And you're storing, logging, tracking all that so that you can make those adjustments over time. Because deterministic matching can be a living piece of your platform and survivorship rules, right?

But one of the things we do is we, so we version our change rules, effectively our transformation code. Although we do it slightly differently in that our context holds the logic and our code's cattle, right? So actually, we version the context, right? This is the logic that we're using, and the code itself just gets generated, run, and then disposed of. But whenever we insert a record or change a record, we tag back to the unique ID or GUID of this piece of logic with this version. So we know what piece ran. Often when I'm working with data teams, and I ask them that question, right, you know, say DBT, right? So how do you know which DBT code ran to populate that record, and they don't log it, right? They don't, like, yes, the dbt code's in Git, hopefully, but they don't append to the record or the log anywhere to say, when we created this row or updated it, it was this piece of logic, this piece of context, this piece of code that did it. And so, I think often that pattern of logging everything is not baked into a lot of people, and they're not taught it. Is that what you're saying?

They're not like audit, any type of audit logs. Well, yeah, yes. Because again, that, that also goes back to that feedback loop. Like, I like to use the example of data teams struggle with metrics to quantify results, right? Of what, what do you measure your data team on? And You know, teams struggle, companies and teams struggle with that because of this specific pattern that we're talking about here, right? They're not logging, they're not saving in a table that is historical and trackable and measurable of what happened, right? Whether that is data quality stuff of this is the number of records that were ingested during this time, this is the number of records that were cleaned out of that dataset, this is, you know, so that you could track those types of pieces over time. This is the feedback loop of, you know, this was wrong, you know, because, and we got this information back from the business, and going forward, this is the new deterministic pattern that we're using, and it improved our data quality by 10%. Nobody has the ability to track that because they don't Because they don't track it, right? They don't log that information. But yes, that 100% is something because it's an afterthought. And it's a team maturity thing too, right? Young teams, small teams don't have the history within themselves to know that that's even a thing to worry about. And they're often In a place where they're just trying to get the next thing done. It's just—

Yeah. Well, they haven't been burnt, right? They haven't been hit by the bus by not logging it. So, there you go. Actually, I had no clue who changed what and what it impacted. So, then if we look at master data management, right, that matching, that survivorship, do you think it should be a system on the side? Like, let's assume we're not going to go buy one, right? We're going to build Because we're a typical data engineering team today. And so, is it a system on the side, right, where data comes in, that system runs, does the matching, survivorship, and then the data is populated and everything else reads the data and pushes it forward? Is it, and the answer to this one's no, is it handcrafted in every data transformation flow, right, the matching thingy? Or is it some kind of context engine that the data transformation flows can call to say, hey, I need to do some matching, almost like a service, right? Hey, this is what I got. Yeah, thank you very much. And then carry on. Which way do you do it? Are you trying to treat it as something that is hydrated in the code every time it runs, or do you treat it as a system that does the work and stores the data?

I like to hydrate it as the data comes, because that makes your code and your platform a little bit more modular so that when you do go from mastering customer to mastering supplier, then it's— you have kind of a single pattern, right, that has helper tables that depending on the domain that's being worked on, that's being mastered, It can look to whether it's that JSON file or I like storing it in tables just because I'm old and a data guy and I like tables. But store those rules in a table again so that you could track it over time so that you can quantify that over time because then that table in itself has dual purpose of hydrating that code as well as being a dimension in the model in some type of query that you're going to use downstream. That's my 2 cents. I think that it's a better fit because otherwise you're basically hard coding that information into your code and it's I'm not a fan of that.

Yeah, I think of it as an engine, right? You kind of pass it over to the engine, and the engine runs it, and then passes you back the result, almost like a service. But there was a good point that you just made as well, was that although you can have a relatively generic engine, the domain that you're trying to master data for actually needs more context. So, you know, if you're mastering people or organizations, you're gonna rely— so let's say we're gonna master people, We're gonna rely on things like Social Security number, driver's license, phone number, if we have to, email, if we really have to, right? There's a bunch of known patterns of what we can use to suggest there's a match, and then the, you know, value or the survivorship value of it.

Right.

Organization, we're gonna look for some kind of government organization number, tax number, that kind of thing. We enter product, it's slightly different, suppliers, their organizations, right? So, Yeah, these kinds of different rules. They're not really rules, are they? Because the rules are the algorithms or the layers that we're running, these different helpers, I think is the right word that you use, right? Helper patterns for that. So, again, it's not a simple problem, right? Master data management is quite complex in itself. Always complex.

Yeah, always complex. That's why we have so many platforms out there, right? that people could buy off the shelf and try to use. And then my experience over the years has been that businesses have a tendency to come back to the build piece, right? Because people seem to have a unique— Yeah.

I think I found that that happens because when we buy a platform, we're not buying a solution, we're buying a tool that can be configured, and actually all the hard work, it helps, But it doesn't outsource all the effort, right? It just assists us. And yeah, I think, yeah, I'm like you. I think kind of building it to a degree, depending on how big you are and all that, that user context, right? And how many of these things you have to do and how complex it is.

Yeah. Yeah, absolutely.

Excellent. All right. Before we close it out, has anybody got any questions?

Yeah, we've got a handful of people. Who's got questions?

Anything we should discuss in the last couple of minutes, given we were slightly late coming on? We'll take that as a no. All right. Well, let's, let's see what the universe aligns with over the next month. And it's always kind of, yeah, it's always interesting when one of the awesome foursome is posting something and I'm like, oh, I'm working on that, or I've been working on that.

It's always funny because I feel like there's cycles to this, right? It feels— sometimes it feels like we're all working on something similar in kind of the same time range. It's funny.

Yeah, I think the master data management one freaked me out though.

Did it?

Because I haven't done an iteration on this for quite a while. And then I just literally doing it, and then— oh, there we go. It's a service alert in the UK. Excuse me, don't know what it's for. Uh, oh, wildfires. Here we go. There's, uh, everybody, there's wildfires in the UK. Everybody's got one. Um, yeah, like context, metric layers, platforms, all that other stuff, that's Still a wildfire in the UK. That's stuff that's been happening for a while. But yeah, the idea that we kind of both stumbled across master data management stuff at the moment as well, that did freak me out.

So yeah. It worked out though. It gave us a good topic for today, for sure.

I have lost Oh, there you go.

I'm back again. You're back.

I'm back. Yep. Excellent. All right. Well, thank you very much and we'll catch up with you in a month's time.

Sounds great.

Excellent. All right. Catch you everybody later.