AgileData #18 - Agile Data Patterns, applying magic tricks you have created before

Welcome to the Agile Data Podcast, where Shane and Nigel discuss the techniques they use to bring an agile way of working to the data world in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Nigel Vining.

So today, Nigel and I are going to have a little chat about this thing we call patterns. And for me, patterns is one of the core things we've kind of developed in our consulting gigs over the last 20 years. And it's something that we leverage a lot in the way we deliver the magic within the Azure Data product. And so for me, a pattern really is something we know we can apply in a particular problem space or in a particular use case that will solve that problem quicker and faster than building something from scratch. So often we can apply a pattern to the wrong thing and make it worse, but when we apply the right pattern to the right problem, we know that that problem will be solved in a certain way, and that takes us less time, less effort, and less risk. Risk to fix that problem. So, Nigel, what's your view of a pattern? How would you describe it?

So for me, patterns is probably— I always think in terms of my consulting toolbox, as it were. And in my toolbox, I know a whole lot of different ways to make something work, rightly or wrongly. Whenever a customer asks me to deliver a piece of functionality or a feature, I automatically go to my toolbox and I identify a pattern that I've used before successfully, and I will apply that to that problem to achieve the desired outcome.

Yeah, so I mean, an example for me might be the file drop stuff that we built. So we have a pattern where somebody can grab a CSV file or a JSON file, they drop it in a place. When we see that turn up, we do something with it. We move it into history in our in our case. So that's a reusable pattern. We know that that allows users or analysts to upload data quickly and it turns up where we need it. And so once we've built that pattern, we lock it, we load it, we make it hard, harden it. And then we know that the question of, I have this data and it's manual, how do I get it there, is taken care of.

Yeah, I'd agree with that. The file drop is a really good example. And even the file drop patterns evolved over time. We started with that pattern, it was quite rudimentary. Then as we went along, we finessed it and finessed it, and now it's a very hardened piece of logic and we know exactly what it does every time and it's repeatable and there's no effort on our part to apply that.

Yep, probably another example I use is the data modeling techniques we use. So in the old days, we would've done star schemas, we would've done slowly changing dimensions type 2. They were patterns that we got from our friend Kimball, and allowed us to say we have some data, we need to structure it in a certain way to make it useful, and that dimensional modeling pattern was the one that we used. These days we use, in terms of agile data, we use more of a hybrid of a Data Vault model under the covers for a whole raft of reasons, but again, it's a pattern. So for us, we talk about concepts, details, and events. So there are concepts which relate to things that we may want to count or manage. So products, customers, employees, stores, orders, payments— they're all concepts of things we need to manage. Um, there's some detail about those concepts, so it might be a person's name or a person's age or the product quantity or the order value or the payment amount or the employee's location. And then the third thing we care about is the pattern of an event. So when we go customer orders product, we know there's a certain pattern that we can use to describe that event and reuse it. So for me, modeling in the data world is a bunch of patterns. And I suppose one of the things we've seen is when the big data bollocks hit, we lost those patterns. We started getting big data scientists that had no understanding of patterns. And so they would just write a raft of black box code that had no repeatability in the way it was written, no repeatability in the way the data modeling was done, and chaos reigned as it tends to do in that space.

Yeah, I'd have to, talking about the event modeling, that's quite a nice pattern for us because under the covers it's a very, it's got really nice structure to it, so when we, the actual coding and DevOps, DataOps that goes with modeling those events is quite a nice repeatable piece of plumbing for us to write because we can write a pattern that deploys those models and then basically reuse it no matter what we throw at it, whether we're throwing customer data at it, product data at it, event data, the model automatically adjusts because it's a pattern. I quite like that one personally.

And I think also the concept we have of rules and the way we use natural language as the way of describing our ETL effectively or writing it or ELT, it's a form of a pattern. So we find a problem with some data, we determine a pattern we can use to fix that problem, and then we make that pattern available as a rule type so that any analyst can quickly apply that pattern to data. So, you know, an example I had the other day, was some of the customer data we were working with, we needed to validate cities. So as we always know, data that's been entered into a source system, or what we call a data factory, it gets manufactured at various levels of quality. And so this data that was in this field called city was particularly unclean and it was fraught. So what we did was we created a reference dataset of all the cities in the world. And then we created a rule type where you could compare the data in the customer table to that reference dataset, and it would come back and flag the ones that didn't match. And then you could take some action. You could fix in the source system, or you could determine some rules where you wanted to clean it up on the way through. So we then baked that in as a rule pattern. So next time you want to do that, you say validate city. What was interesting is because New Zealand's address structure is kind of funky. A lot of the data that was in the city field for the customer was actually around suburb and location. So what we did is we extended the pattern for that validation out. We went and grabbed a list of all addresses out of New Zealand and brought that in as a reference dataset. We tagged each of those types of address information, such as suburb and locality, as a bunch of features. And then we updated that rule type to say validate city, but as well as validating against the list of cities from the world, also include suburbs and localities out of that New Zealand reference dataset. So again, bit of work for us to bake that pattern in, but next time I do it, click of the button, I can say validate it against World Cities or extend the validation out to these, this other reference dataset. And so again, I've got a problem, I have a reusable pattern for solving that problem. I can apply that pattern in less than a minute and get the value from it.

I was just trying to think of an analogy. That's a good one. Build once, reuse many. That's where patterns, I guess, for my consulting background, patterns are nice because you can quickly apply them to the problem at hand to solve something.

Yeah, I remember really in the early days, we, you know, in the consulting market in New Zealand, there were a bunch of experts and it was back in the days where we used to use diskettes, so 3.5-inch disks which we used to carry around before USB drives were a real thing. And there were a bunch of people that had some code on those diskettes that they effectively kept in their top pocket. And so if you had a problem and you knew the person had a patent to apply to that problem, you had to hire them and they'd bring their little diskette, their code, they'd fix the problem for you. There was always an argument then about did the customer own the fixing for that. And so actually the accessibility of those patterns was pretty limited. And I think we've seen a lot with open source now, things like Airflow and those kind of tools where a lot of the patterns are being shared. And so for us, what we want to do is remove the complexity of applying those patterns and make it a one-click magical experience. So have this problem, click a button, it solves it for me.

I have to agree with that one. I used to carry around, I guess it was a CD at the time, and on it, it was literally a CD that was filled with SQL scripts and stored procedure scripts for Oracle. And at each gig, you would effectively chuck your CD in, grab your patterns out, and change the names of them and apply them for the customer's environments. But then as Shane said, Patterns are now accessible to everyone. A lot of them are open source, people freely share them. You know, it's very democratized how patterns get used now.

But also the idea that every time we touch a piece of customer data or every time we work with a new customer, we get given a new problem we don't have a pattern for, or one we haven't actually made into a rule type. So that's what's really cool, right? As you go, well, actually there's 101 problems we could solve. And actually, if we guess what the priority is, we'll probably pick the ones that may or may not be common. So there's a bunch of patterns that we've applied from day one because we knew they had value. But now it's a case of every time we strike a new problem with customers and their data, we look for the patterns that we can make reusable to everybody else. I think it's also important to say that patterns aren't only around technology and code. So, you know, we have this concept of an information product, which is a pattern of how we can engage with business owners or business stakeholders and very quickly understand the core business questions that they want answered. So, you know, how many products have I sold? How long does it take to go from selling it to get paid? Things like that. And then also the events that make up that, which is the data that we need to be able to answer those questions. And so That is a pattern we use repeatedly when we're talking to people to understand what they want, but it's not code, it's not technology, it's just a way of solving the problem, which is how do I suck out of the brain, you know, my stakeholder's brain, the things that are really important to deliver first? Because from an Agile point of view, we want to deliver the highest value piece of work first.

Yeah, so those, I guess, it's a people pattern that it's effectively, it's not code, but it's a standard list of questions like a checklist checklist that we work through with new customers to effectively get the things we need to know up front and take them on the journey to elicit the responses that, you know, we know we're gonna, you know, need to get from them.

And I think one of the other lenses to look at when you talk about patterns is the inappropriate use of patterns. So the one hammer to, you know, for every nail, because it is tempting when you have a certain way of of working or a certain pattern that you know can solve many problems, that whenever you see a problem, it's like, well, can I use that pattern on day one rather than go to all the effort of investing in a new pattern? And so you've got to be a little bit careful about, you know, again, data lakes is my classic example. You know, data lakes have value. The ability to store structured and unstructured information quickly in one place to be able to use it is a pattern that has value. but the idea of using a data lake as the only way of managing your data is just ridiculous. And so I think we've been through this phase of one pattern to rule them all. And so now we're moving into the land of hybrids. I was watching, listening to one of the excellent webinars from Eccleson, because they're one of my favorite analyst companies in terms of providing insight into where the market's going and what's happening. And they were talking about a data lake house. So the combination of using a data lake to get data landed very quickly and then moving back into some of the more governed patterns, dare I say, at a modern data warehouse to fix the problem of ungoverned code and ungoverned data and lots of people writing their own stuff, using their own patterns, and then stakeholders getting 25 different answers to the same question. So we've got to be a little bit careful of overusing patterns where they don't fit, I think.

Yeah, that's an interesting point, actually. That almost— what came to mind was an anti-pattern for me. Recently, an analyst I was working with, his background was obviously in a traditional relational database, and he'd come up with some beautiful code to produce an answer. And then what he'd done is he'd taken his pattern and applied it to a modern massively parallel database centers. It just literally didn't work because his paradigm was wrong. His code was great, but it was never gonna run on this new platform because it was what we'd call an anti-pattern. It didn't take much to effectively explain to him why it didn't work and how he should adjust it, and he did, and it worked fine. It flew. He got his results in literally minutes, whereas previously it'd run for over an hour. Achieve nothing. So that's where you, I guess, say a pattern that's in one environment becomes an anti-pattern in another.

And for me, I kind of got to the stage now where I talk about technique. So from an agile data point of view, I really like the analogy of cooking and food and restaurants and the skill in doing that. And so if we talk about you have some technology and that's really the utensils and the equipment that you have in a kitchen, And you have a bunch of ingredients, which is typically the data that we have to play with to produce a meal or produce an answer to a question. Then our ability to use techniques with both those ingredients and those tools or equipment, that is the combination that gives us the value. So, you know, if I had a cast iron pot and I was trying to make a béchamel sauce, it's going to be very different than if I've got a wok. right, or an aluminium pot or a barbecue. So if I had excellent technique, I probably could make that sauce in a wok, which I don't, but I probably need to move towards one of the tools or the equipment that's more fit for purpose. And with the amount of new technology and variability we have in the data space these days, us applying the technique that we knew worked for us in the past with new technology is sometimes an anti-pattern. So I'll take that example, you know, you take it back to that person, he had great skills, he had great technique with certain types of databases, but when we apply that technique to the new types of databases like BigQuery, you know, it doesn't optimize. It'll run, but it won't run well, you know, it'll be a good, it'll be a source, but it won't be a beautiful smooth source. So I think that's important is understanding the combination of your skills, your techniques, reusing those patterns, reusing that equipment to get the best outcome.

Yeah, no, I like that analogy. Yeah, no, that's very appropriate.

And so for me, what I think about now in terms of those techniques and patterns is how do we teach them? How do we coach them? So we have a level of maturity of those things. So if I can explain it to somebody else and they can use that pattern and display that pattern using a technique the same way I do, then that's a really hardened pattern, right? It's mature. I can teach it. Before that, you know, I'm probably requiring an expert who understands all the complexity and can take what I'm telling them and apply it. And before that, you know, we're starting to get down to not a novice, but somewhere in the middle there where, you know, somebody could potentially follow that way. And then ideally, you know, we've got a novice, if we can teach a novice that technique or let them leverage that technique. So going back to that City, if a novice could say that column's called City, apply this rule, tell me what's wrong, that's really the nirvana of being able to empower analysts to apply those techniques in a magical way, which is ideally what we're after. So One of the things I struggle though with is if you have somebody that's at an expertise or a coaching level, they often don't want to adopt other people's patterns or techniques. And that's a really interesting experience. So, you know, would a hardened data engineer use our platform to change their data because we are faster than they will ever be and So far in the experiments we've done, the answer is no.

Yeah, I think there's a— I think a lot of people fall back on what they know, and what they know is they have their own toolkit and they grab their code out like your floppy disk example, and they want to write their ETLs or ELTs or their transformations, and they want to write them by hand because it's what they've always done. They know it works. They know it takes a long time. They know the testing of it's complicated, but it's a comfortable known pattern. I think even though you can remove all that complexity and, you know, using a robust pattern like under the covers of agile.data.io, people are, you know, nervous, I guess, to deviate from what they know.

Also because it changes the way they get paid.

Yes, you're right, because you're effectively saying I can now do in 5 minutes produce all of your transformational logic or your patterns. I can produce them in 5 minutes using this template, or you know, you can spend a week and hand code them. So that's, it is undermining how you get paid per hour.

And so when we look at where technology's gone in the data marketplace, we see we've moved away from suites of products back to best-of-breed silos. So if we want to look at how we collect data out of those data factories, we need some patterns and techniques which require a certain technology stack. When we want to land it or put it down somewhere so we can use it, we need some more techniques and patterns, so another technology stack. Where we want to load it into some form of data storage that can change the way that data looks, we need another one. And then when we want to write code that actually makes those changes, we need another one. When we want the catalog to understand, users can understand what we hold and what it means, we need another one. My view is over the next 5 to 7 years, people will get upset or have had enough of cobbling all that best of breed back together and we'll move back to suites because we always seem to go in a 7-year cycle. So it'll be interesting to see how much when it happens, whether this concept of a pattern becomes more prominent or whether it's again bake your own within suites or within the best of breed.

Yeah, I agree, the next 5-6 years will be interesting. I guess there's a certain level of dissatisfaction with the amount of integration that seems to be required for a number of the new, I guess, best of breed platforms because they are literally just a platform and there's a lot of integration required to make them do smart things.

And again, it's just a pattern problem, right? So, you know, we know that, uh, dropping a file into Google Storage and then moving it into BigQuery, which is what we do— we have a bunch of patterns that do that for us, uh, so we don't have to care. Um, you know, if we wanted to, dropping data into an S3 bucket in AWS and a pattern to bring that into BigQuery again, uh, it's a slight different technique, but it's the same pattern. So if we wanted to, we could apply a pattern model to allow us to integrate these piece-to-brief products. But that's not how the marketplace seems to think. It always seems to rely on consulting companies and technical people to spend weeks, days, years to do that integration, and every time it's a bespoke thing. So ideally, the market will move to more patterns. One of the things I still struggle with is how do we write patterns down in a way somebody else can read them, understand how to apply that pattern and which techniques they need, and more importantly, understand when to apply them and when not to. So that's something that I think we probably need to work on a bit more, is explaining those patterns, because the patterns are valuable and we have no problem sharing those patterns. We just struggle with ways of being able to do it without me talking and waving my hands. So hopefully we'll get better at that in the future and we'll publish more of our patterns out so we can add more value to other people.

I look forward to that.

All right, well, that's Nigel and me talking about patterns. We'll catch you next time.

Thanks, Shane.

And that, Data Magicians, was another Agile Data Podcast from Nigel and Shane. If you want to learn more about how you can apply agile ways of working to your data, Head over to agiledata.io.