Data Engineering Patterns with Chris Gambill - Episode #65

Welcome to the Agile Data Podcast, where we talk about the merging of agile product and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, "Hey, that's not what I wanted," and then you hear your team reply with, but that's what you asked for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the green book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

I'm Chris Gamble.

Hey, Chris, thanks for coming on the show. Today we're going to talk about data engineering patterns. Before we rip into that, why don't you give a bit of a background about yourself to the audience?

Absolutely. So, I have spent the last about 25 years in the data world. I actually got my start in a call center back in 2000, where I quickly became the kind of go-to person for untangling all these Excel spreadsheet nightmares and all these manual processes, and they figured out that I had this knack for automation and data, and I quickly got pushed into a position that was what I did all the time. And since then, it's really taken off from there. I've worn a lot of different hats from individual contributor all the way through director positions. I was brought up in the Microsoft stack, so a lot of DTS packages early on, SSIS. Moving into Azure and ADF and Synapse and Fabric now, and then even had opportunities to touch Snowflake and Databricks and even do some migrations from AWS to Azure and back again. A lot of kind of breadth of knowledge as well as really deep knowledge with the Microsoft stack. Most of my knowledge has been self-taught and really tested on real-world projects, right? And then today I run Gamble Data where I help organizations design scalable modern data solutions, automate reporting, and really bring a lot of order to the data chaos that you see out there, right? And I'm also recently proud to say that I am now a Microsoft Partner as well. Super excited about that. And that really lets me to stay ahead of the new technologies because I get some advanced views of different things that are coming out from Microsoft. Super excited about that.

Excellent. Hey, at some stage you might take a sidetrack into, is Fabric more than just PowerPoint slides with Synapse renamed? But before we do that and Before we get into the sarcasm from me, why don't we jump into the core of the subject? I suppose if I put some anchoring in place, the way I think about patterns, and I get it out of the pattern book that came out about buildings, I think about them as solutions for common problems, which fit a certain context. So I can look at the way people submit code to Git and say, hey, that's a repeatable pattern. I can look at the way people pair program and say, hey, that's repeatable prep. Pattern. I can look at the 5 ceremonies of Scrum and say, hey, that's a repeatable pattern. I can look at a 4-tier data architecture and say that's a repeatable pattern because each one of them are a solution to a common problem. And given your context, it may fit, it may be valuable, or it may actually be an anti-pattern. So let's use that as the anchoring and what's one pattern that kind of comes straight to mind in the work that you do?

Yeah, so really when I got a chance to work with AWS, this is one of my favorite patterns out there, partially because I think that this pattern gives you a lot of control as to what's going on with your whole ETL/ELT process. And that pattern is to write a Python script to do all of your extracting, your transformations, and your loading. Take that, containerize it using Docker, create that environment in Docker and really containerize it, and then take that, and AWS has some wonderful elastic resources where you could load it into an elastic repository in AWS. And then from there, in this example, I generally would use Fargate and I'd run them in the elastic services and Fargate. And then we orchestrated that with kind of cron schedules, because most of that was batch scheduling. And that lends itself well to batch scheduling that you're going to do maybe once or twice a day, bigger processes, because if you're doing any thing more often, right? Like 15-minute low every 15 minutes, or if you're doing any type of streaming, probably need to go the Lambda route. But with Fargate and this particular pattern, it lends itself to that. Otherwise, the streaming, you'd go a little bit different direction because cost can be high with Fargate, but Fargate does really well with this specific pattern.

And the thing that I like about the way you described that is I can think of them as Lego blocks. So there's a bunch of Lego blocks and you've racked them and stacked them together, and then you've described the anti-pattern. So if you're doing batching, if you're doing 15 minutes or more as the frequency of loads, then this is gonna work. But if you go into anything below 15 minutes, definitely near real time. But even if you went down to a 5-minute load, you're gonna have a whole lot of things you need to deal with, and this may not be the best pattern for that. It would be a more streaming-centric type of pattern. Is that how you think about it?

Absolutely.

Absolutely. Yeah.

'Cause then you need to probably go the Kafka route, or like I said, Lambdas, and go that direction.

And then often there's patterns within patterns. So for example, when you talk about this extract and load pattern you've got, and you talk about effectively you containerized it, then I'd be looking and saying, okay, are those containers running 24/7, or have you adopted a form of a serverless pattern? Are you effectively deploying the container, running it, destroying it, deploying it, running it, destroying it? So there's always patterns within patterns, which make it a little bit trickier, but What's your common one for this? Do you tend to run that container permanently or do you adopt a deploy and destroy kind of pattern?

In these cases with the batch loading, it's usually adopt and destroy, right? It's usually you spin it up and then you're killing it at the end because that Fargate instance dies at the end of the run anyways. So it's a spin up and then burn down at the end.

And then the other one that's interesting for me is that you're pushing it to Elastic. So that's Elasticsearch, yeah?

Not Elasticsearch. So Elasticsearch is a little different. So in AWS, there's a couple of resources called— it's ECS and ECR, right? One is the repository, obviously the ECR piece, and that's where all your containers get stored. And then you have the Elastic Services, and that is where all the scheduling happens, where you're defining whether you're if you're going to spin up an EC2 cluster or if you're going to spin up a Fargate cluster, right? And set all your permissions and your security pieces and all that.

Okay. So that's almost the orchestration of how you deploy and destroy those containers. Is that right?

That's correct.

And this is the thing, because I only work in Google Cloud Platform typically these days for our product, I'm trying to map your patterns.

Yeah.

So I'm going, oh, okay. So effectively we run Cloud Run and Cloud Build. That's the equivalent of the containerization. We run Python when we have to. Yep, that makes sense. We only use it when it's 15 minutes or more because we have a whole lot of other patterns that we need if we're going to go to anything that actually transforms faster end to end. And then for me, it was like, oh, the Elastic thing, right? I just naturally went to Elasticsearch because that's the last AWS thing I used that had the word elastic in it. Again, I think one of the tricks with patterns is explaining and clarity what they are, and then actually explaining the context, explaining the anti-patterns is how I tend to find the easiest way to do it is, we know if it looks like streaming, this pattern may not work.

Yeah.

Find another pattern. Otherwise, you've got to go and actually do a whole lot of detail and describe it in words, and that's actually really hard. So how do you do that? How do you actually document a pattern that you've used so that you can remember it next time or so somebody else can use it?

So when I was at AT&T, for example, I would create these patterns and I would actually document all of it a couple different ways. One, I would write it all out just in a Word doc, save it on— in a— we had a SharePoint site. This was 10 years ago. I don't know. Time gets away from me. And so save it in SharePoint. But I would also create PowerPoint presentations of the slides of all the different steps so that people could see, okay, these are the resources that you need for this pattern. This is the description of the pattern, then step by step, mostly because people like the step-by-step-ness that you get and feel that you get with the presentation, as opposed to just people glaze over when they're reading a Word doc.

I suppose, and that step-by-step means, especially if you put screenshots in, it gives it more context. People can just go through and see what you're doing and that gives more context or follow the steps and learn by doing, and that makes sense. And then if I think about what we do, because we have a similar set of patterns, completely different technology stack. One of the things we do is we tend to deploy or land all our data in Google Cloud Storage. So equivalent of AWS S3 and what's Microsoft this week?

It's, yeah, Microsoft has Azure Data Lake Storage.

And then there's Gen2, right? And then there's OneLake seems to be the new fabric. This idea that we're actually landing the data into file-based storage. And the reason we do that is because we have a pattern where we load from Google Cloud Storage into BigQuery. And if we load from Google Cloud Storage into BigQuery, it's free. We don't pay any compute.

Oh, okay.

If we push the data directly to BigQuery, we pay for the compute processing for it to actually store that data. Now, if we're using Google Analytics, so GA4, there's an out-of-the-box adapter that goes GA4 into BigQuery. So we bypass our GCS layer. And then if I think about out-of-the-box adapters, and this is where I'm going, we'll write a Python adapter effectively when it's a system that's not well served. So what we find is with every organization we work with, they'll have 3 to 4 of the standard systems. They'll have a HubSpot or a Salesforce, the ones we've seen before. and we can use an off-the-shelf adapter for that. And then they will always have one system we've never seen before. There is no adapter, only 100 people in the world use it, and we actually have to write the framework to collect that.

I was going to say, or even worse, they have a very custom product that they've created in-house that you have to figure out the create your own adapter. It's a BYOA, bring your own adapter.

Yeah. And of course it's not documented. There's no schemas. There's sometimes no API. You've got to go into either the database or the logs. Actually, I don't mind I don't find that one so much. I find the one that does my nut the most is they're using an off-the-shelf package, something like Salesforce or HubSpot, and then they've customized it so much that the off-the-shelf adapters don't actually work. So now you're actually custom building an adapter. And there's a bunch of companies out there that do nothing but build adapters. I'm thinking Fivetran, Data DDO, Istri, those kind of companies that do nothing but that.

Yeah.

But let's go back to that case. So let's go and say we have a system that you're going to need to collect data from. It's a well-known one, so you can use one of the commercial software as a service products to do that collection. So effectively in my head, you're replacing the Python script part of that pattern that you talked about. Is that how you think about it? Do you actually plug and play to a degree? When do you decide that component of that pattern can be replaced with another one? And how do you decide which parts of your pattern survive? So which ones become almost pets and not cattle, because that's always a bit of a trick.

It is. And I think to your point, if it's a very standard implementation of something that has a plug-and-play pattern, a plug-and-play adapter in it where you could just pick up like ADF, right? And there's an easy connector, it's straightforward, then you could swap it out for those low-code/no-code pipeline builds that you could create. When I was in cybersecurity, there was some security involved. And there was some lack of trust in the low-code, no-code tools. And so the Python was the way to go. And just because, again, we had a little bit more control when it came to security and we needed that from a governance standpoint too. And so those are the types of things that you have to take into account when you're looking at which tool you're going to use. Like Informatica does such a great job from a governance standpoint when it comes to that type of stuff, depending on how regulated your domain is.

Yeah, did until Salesforce bought it and then had a data podcast recording in 10 seconds for that. I come back to this idea of if you think about a pattern being made up of a bunch of components, each one of those being a Lego block, and you're building these Lego towers that have fit for purpose. The context again of that end-to-end pattern, but also the context of each component is really important. Because if I dropped into an organization and they already had the extract and load patterns in place, it was the one that you described. And then somebody says, oh, just go hit Salesforce or HubSpot. I'm naturally going to go and want to look at a commercial adapter because I go, I don't want to write that. The frameworks are all out there. They're robust. They're relatively cheap compared to the human effort of writing one and maintaining it. And so I'd naturally go that way. But if the context of the organization is all data must stay within their boundary and they treat their boundary as a cloud boundary as well, then actually there's no point. I can't bring that adapter in because the context of how we're choosing these patterns says that's actually not the way we can do it. And that makes sense. So context is king as always.

Absolutely. Absolutely.

All right. So that's extract and load. Hit me with another. Okay.

So another one is Databricks, right? So just to give you a glimpse of kind of what happens in Databricks for this specific pattern. In Databricks, you could choose between two types of compute. You have dedicated and you have serverless. And with the serverless, you can't just create an initialization script that loads all of your libraries that you need into the compute, into the environment, because it gets respun up every time something happens. You know, that's the point of having that serverless. Where with the dedicated, you can preload all your libraries into that environment and you're good to go unless you have to reboot that environment. But then you have the init script that reruns everything. So just an overview as to what's going on here. So with the serverless, the pattern that I use because of this is I'll create a bootstrap notebook that goes through and says, okay, if these are the libraries that this notebook needs to install, then this is where it goes. Because if you have a wheel in place, it is easier and quicker and from an efficiency standpoint, a little bit better than having to go out to PyPI to download and install that library. So I have a class and a set of functions that goes through and says, okay, this is the library that needs to be installed. It determines whether it is one of the ones that we have a wheel for, and those wheels are generally stored within— in this specific pattern, we're talking Azure, but it could be, like you said, with the Lego blocks, could be easily replaced with AWS S3, but this one's ADLS. And so it goes into ADLS, grabs the wheel, does the python -m pip install of that wheel, and then moves on with life. If that wheel's not available, then it'll roll to the PyPI version and download and install the PyPI. But it consistently does the same thing across that whole project's environment. And it provides you with one central location for maintainability and for consistency across your full environment so that when you're troubleshooting, you have one place to go.

Okay. And so this is where you're getting technical and I probably need my co-founder Nigel, who's the technical one of the two of us. Well, really technical. He's the engineering one. But let me play this one back and see, because this will be a good test, right? If I understand the pattern, even though I don't understand the technology. So what I think you're doing is I think you're templating the initialization of the serverless component. So the serverless component is ephemeral. You use it, it deploys itself, it destroys itself. So unlike a container where we say go and then die, with serverless, we effectively say run and it's doing that for us. Is that right?

Yes, exactly.

When we deploy a container, the initialization of all the things we need are embedded within that container. I'm old, so VMs were a thing when in my days. So we're effectively deploying this container and the initialization and all the things we need are embedded in that container. So we don't really have to care. It's like container deploy, run, container destroy, and we are in charge of that. That's right. That's the pattern for containers.

Yep.

Okay. And then with serverless, Effectively, they're not serverless because there is a server and there is containers. We just don't maintain it. We just go run this thing and it knows to hit that server, start up that container, run it, kill itself. Yeah. So that's the serverless pattern. And then what you're saying is if we tell it when it boots up to then go and deploy all these scripts, it works, but it's not optimal. Whereas if we effectively have a template and we say, whenever that service is invoked, go and use this template, we get more control. Is that what you're saying?

So with serverless, every time a new orchestration runs, it's almost like you're deploying a new empty container. And with containers, with like Docker, you're telling— within that Docker script, you're telling it, okay, these are all my requirements. It provides this initialization script that runs throughout those requirements. With Databricks serverless clusters, it doesn't accept that. There's no ability to do that. So when your orchestration notebook kicks off, if it's not one of the pre-built-in libraries that Databricks has loaded into its serverless environment, you have to install it. You have to do either do a pip install. With Databricks, there's this magic key, right? That's a percent symbol and you could say run and then basically do a command line command there, and you could do a pip install, but then it becomes inconsistent across your environment, right? Because every engineer that goes through and creates a notebook might do it slightly differently. They might not know that there's a wheel or not. They might do a magic and the run and pip install, or they might do a command line and say python -m and then do it that way. So to provide consistency every time that those notebooks kick off, then we have this kind of bootstrap notebook that does that and provides that consistency and maintainability across our whole serverless environment. Does that make sense?

Yeah. So, if I think about this idea of a data platform that has a serverless component and let's call it a persistent component, It reminds me of one of the many iterations of Azure where we had Synapse Serverless and Synapse, what was it, pool, the old PDW stuff.

Yeah.

Yep. So that pattern of a serverless component and a persistent data component seems the same pattern in Azure with Synapse as it does with Databricks. But I think what you're telling me is if I was going to deploy this pattern, it works for Databricks because Databricks technically works slightly different. But would it, would I do the same thing in Azure Synapse, or does it actually allow me to solve that problem by just telling the serverless component to initiate these things?

The nice thing about Synapse is that Synapse, when you're deploying a pool, you could tell it and give it an initialization kind of script, right, of these are 'Python libraries that I need loaded for this,' whether it's serverless or dedicated, whichever the pool you decide. But I think the Spark pools there are all serverless. You could predefine it, but Databricks doesn't allow you to do that. You have to either do it in every notebook that you create that's going to run, or you create this kind of bootstrap script, that notebook that does that, that you basically are just appending onto your notebooks that you create for all your other ETL jobs.

And that's key, is that the technical deployment of a pattern may change depending which technology you use, and there's always something slightly different, which is why they have competitive advantage or not. So you gotta be really clear about that. When you pick up this pattern and you're putting it into a new place, you need to actually understand there may be something I need to technically tweak. And also over time, things change. So for example, we used to use Cloud Functions for our transfer transformation code. So we would use a deploy and destroy on our transformation code as well. So effectively it would read our context engine, it would work out what it needs to do, it would write the code effectively or hydrate it because it was standardized, it would deploy it to a Cloud Function, it would run, it would then finish, and then we'd go on to the next one. So we're daisy chaining these little blobs of code, but the manifest of how they built were dynamic. Back then, Cloud Function would only run, I can't remember, for a minute or 3 minutes. So you get these ones where if you were waiting for it, it would just terminate. We had to do a fire and forget pattern. But then Cloud Functions went to 9 minutes, I think, and most of our small chunks of code would run within that. So then we had to re-engineer the code to just use it. And then we had to move from Cloud Functions to a whole lot of other new things because one of the downsides of Google is they love to deprecate their products whenever they feel like it. So the point there is maybe the core pattern itself isn't changing, but the technical implementation of it needs to be iterated over time because Data platforms and technologies are moving at such a pace that actually you always have to go back and review your technical patterns, don't you, to make sure they're still the most efficient way of doing it?

Absolutely. And from my perspective, I try to go through those patterns at least once every 12 months, if not every 6 months, because to your point, things change so incredibly frequently, especially if you take Fabric, for example, right? Fabric 18 months ago is not what Fabric is today. By any stretch of the imagination. I initially looked at Fabric and was like, this is a marketing tool to try to push people into some new environment and try to raise their cloud bill. In fact, that's 100% what it felt like when our Microsoft rep came and talked to us about it. And I was like, how does this save us money? He's like, operationally, we'll save you money. And I was like, but can you give me a cost difference between what we're paying now and what we'd pay on Fabric? And he's like, you're not going to see a decrease there. You'll see a decrease in man hours. And so I was like, okay.

Yeah, I remember I got asked to come and do an architecture review for an organization that was deploying Databricks. And somebody obviously came into the organization and said, let's go Microsoft Fabric. This was quite a few years ago. And so engaged with the Microsoft team, because here in New Zealand, they're a great team.

They're really helpful.

And said, yep, show it to me. And we had nothing but PowerPoint. And you sit there going, because I used to work for Oracle many years ago. So I'm like, I did a lot of PowerPoint demos back there. Been told that the product was real. But I suppose my view back then was this is going to go either one or two ways. This is just going to be marketecture, right? They're not actually going to do any changes under the covers. We're just going to rebrand them and recobble together all the quite disparate technologies that had happened by that time. Or they'll Power BI it and they will actually re-engineer it over time and they will make it cheap as chips and they will completely dominate the market where you're a Microsoft customer, why you wouldn't use Fabric. So I think we must have looked at it back then at the same time when it was just PowerPoint slides, but you're saying now it has changed. You're saying that actually there's some meat and potatoes under the PowerPoint now.

I think especially now, I will say that Microsoft right now is definitely coercing people towards Fabric in many different ways, but it's also grown a lot. Yes, absolutely. At one point, what was the phrase that you used? Something about marketing. Yeah, architecture. Yes, I love that. Okay, I might— I'm going to steal that. At first it felt like architecture, right? It 100% was, let's take ADF, let's take Power BI, let's take this OneLake that we just started implementing with Power BI, and let's put it all within this one basic GUI, right? And that's really all it was. But now they allow you to spin up a SQL database, and that's relatively new in the last 3 or 4 months. They have this new version of ADLS that is OneLake, and that kind of helps. And the interactivity between all of it, because it sits physically closer in their servers, then it runs a little bit faster. So yes, from an orchestration and from an operational standpoint, it's a little bit more efficient. And the cost structure has significantly improved. When it first came out, the cost structure was, frankly, in my opinion, a little bit ridiculous. The minimum was $10,000 a month or something like that. And it was nuts. But now you could spin up like an F2 and it's much more reasonable in cost than it was 18 months, 2 years ago.

And one of the interesting things about that is if we think about technical patterns and actually Roland Voss and Dirk Lerner got a book coming out this month. In it, they talk about design patterns, which are the really big Lego blocks, the core meta patterns to a degree. And then solution patterns, how you can actually use them. And then they've actually got technical patterns where they push code into Git to show you how it works. We can look at vendors and we can see patterns. So I'll give you 3 examples. So Microsoft, what I saw them do with Fabric was a couple of things. They started focusing on the control planes. As you said, they focused on how do we have one UI that can actually orchestrate or help us build and orchestrate everything? How do we have one lake? How do we have one place where there is effectively a catalog, a Unity catalog, that has a control plane where all pieces of data go through, even just from a contextual metadata point of view? And that makes sense, right? That's like getting the foundationals in, because really, if we got to the stage where I could use that UI or those control planes and It goes and determines which of the many technical components it's spinning up, what's the best, most cost-effective and performant thing for the job that's about to be done. And I don't have to care. That's a great pattern. And then the second thing that you talked about is because they were early, they need some early adopters and they need people they can get good feedback and they don't want to open up to the entire market because the product's not ready yet. So what they do is they put a price on it that only certain large customers can afford. To stop everybody else using it because they know it's not quite ready. And I'm not taking a potshot at Microsoft. That's a great pattern to deploy and pivot your product. Here's another pattern that we'll see in the next week. Databricks and Snowflake have gotten into a habit that they did what used to happen when I was at Oracle. They will announce features that don't exist now. Yeah. At both their summits, they will announce features that may turn up in 12 months. And then they have a pattern. Yeah, it's, ah, they all use different words, right? But there's the internal one, and then there's the small bunch of really trusted users, and then there's the slightly broader, but you still have to be invited, and then there's the early adopters, it's open but don't trust it, and then there's the, this thing's scalable. It's been tested by their customers a lot. And that's normally a 12-month cycle. So when we see these lovely architecture announcements right now, while I get grumpy with them, I look for the patterns of, okay, where in their deployment cycle is that piece of code? How does it fit with their strategy? And then the last one, because it's topical right now, is Fusion from dbt. I remember when I was, again, Oracle, I'm sure we had a product called Fusion, and then it got called Confusion, which I see a lot of dbt competitors now starting to use. But you think about their problem, right? They have open source product they're not making money on. That everybody uses for free, that everybody expects them to keep developing and giving them features for free. Yet they are now a commercial company that has VC funding and actually has to get growth or profitability, right? That's the two things they have to do. I don't see why the market's surprised that we're now merging those two things and making them more and more paid. That is a common pattern. Yeah, I mean, it could be worse. It could be Elasticsearch, right? AWS picking it up, calling it an AWS product, and not actually letting dbt get any of the revenue. I'm not sure that's a better plan.

It might be Salesforce's next acquisition.

What, dbt? Oh no, they need a database, right? Why? I was thinking about the other day, Teradata. Salesforce has to buy Teradata.

Yeah, because that's the next tool that we don't really love using, right? Is if they went through and they— Salesforce bought Tableau, which is the last generation of BI tools. Like everybody uses Looker or Power BI or all these other visualization tools. And they're like, let's buy Tableau for a huge amount of money. And then they're like, okay, let's buy now Slack for a blue billion dollars. They bought Slack, they bought Tableau, and then now they're going to buy Informatica. And these are the stepchildren of the tools that we love using. And so Maybe like you said, Teradata feels like the database of choice for our stepchildren of databases that, that—

Oh, especially cuz Teradata actually replatformed their on-prem database to be cloudy and they brought their price point down. They just missed the market by a couple years. So if we look at patterns, let's look at the Lego blocks of an end-to-end platform, communication, Slack, visualization, Tableau. They've got their AI thing, right? They've now got the transformation and the data acquisition. They've got MuleSoft. They've got the streaming version of that. They just now need storage and then they become end-to-end. Yeah, it's a pattern.

They become the Oracle of, of today, right? My, yeah.

Although I do remember my theory back then when Oracle was buying everybody was they bought their competitors because they couldn't win against them. The reason I moved into data was I started out in ERP and financials and we used to get our asses kicked by PeopleSoft all the time.

Oh yeah.

Yeah. Like just every deal, it was like our product was ugly. and their product was beautiful. It was like, it was as easy as that. And then my view was Oracle bought PeopleSoft to kill it. It was, we can't beat you, so we'll buy you. And that was actually a hostile takeover. Hostile takeover. Yeah. Poison pills and cool names like that. Anyway, let's go back to technical patterns. Oh, before we do that, actually, what the hell is a wheel? A wheel.

Okay. That's a good point. So there are a couple of different ways that you could install libraries, Python libraries. One is you could go out and basically you're gonna phone a friend, right? You're gonna call PyPI, you're gonna call this PyPI, which is this big repository of all these Python libraries, and say, hey, this is the library I want, I'm gonna download it and I'm gonna install it. Or you could create a wheel, and a wheel is basically a local file of the library that you could save them locally on their computers. They, you could save them in an S3 bucket or an ADLS container. And then that way you have a static install of that library. I think it's probably the easiest way to explain it.

Okay. So it's like in my head, it's what we call a manifest. So we effectively go and grab all the transformation code we need for a job. We then basically build out that Python or SQL series of steps. We then keep it as a manifest and then we run that manifest and then we destroy it. But what you're saying is it's basically a manifest. It's a list of instructions that are stored somewhere that you can make it reusable. And it's, so is Wheel a Databricks term?

It, yeah, no, Wheel is, Wheel is just a Python like it's a, it's like an executable for Python libraries, right? It holds all of the instructions for that specific Python library that you want to install. And the advantage of having a wheel that you use to load every time is that one, you're not having to call and download that file from PyPI, right? So it's a little bit quicker, more efficient, but also you have a consistent version of that library. So every time it— if you call Py, it's going to give you the latest one every time, which can cause issues in your ETL if there was some major change in the version, in the updated version, right? And so you download and use these wheels to do your installs and that way you wait until you're ready to install the latest version and you've done any type of regression testing and made sure that everything's gonna run correctly on the new version, then you could download the new wheel and then save your wheel to your blob container, wherever it may be, and then reuse that.

And that, again, it's a pattern. You've just described a small pattern that solves a problem with a context. And one of the things about patterns, even we're talking design solution or technical patterns, the language changes and it's all about context. So the example I use regularly from that book about buildings is if you have a lounge, a sitting room, and you're in a sunny place, you're typically going to have big windows. If you have a bathroom, you're typically going to have a small window, and it probably is going to be opaque. I've got a friend that lives in Iceland. Their windows are very thick because of the cold, so their design of their houses are very different. Got another friend that lives in one of the most beautiful spots in the world, on effectively a hilltop, and Nobody overlooks them. So their bathroom actually has a shit ton of windows and they're not opaque because they want light in their bathroom and nobody is around to actually see in, in theory. So if you think about it again, that context is key. In that example you gave with a wheel, when does versioning and making a static copy of those libraries have value versus when do you actually want to reach out and grab the latest version? It's just context and choices. And every pattern is important to do that. And language is important. Excellent. You got another one?

I do. I have a third one. And this one is more Azure-based. So it's more that kind of low-code, no-code, because we're going to use ADF. And in Azure, one of my patterns that I use for, or even orchestrating in this case, pipelines in Azure Data Factory and even Databricks libraries or Databricks notebooks is you can actually orchestrate all that with an ADF pipeline. Which is really cool, which is nice. You don't have to go out and do an Airflow thing. So if you're in an environment where you don't have access to Airflow, then ADF provides you availability of this pattern. And so basically you're creating a pipeline where you are dropping in these tasks into your pipeline that are run notebook. You could run it like I said, you could run Databricks notebook, so you'd run Databricks notebook and then you could drop another task in that's run pipeline. And you define which pipeline it's running. If it succeeds, then it moves on to the next pipeline that it needs to run within that orchestration. And then if it fails, then you could, then you're going to, and this is one of those patterns within a pattern where if it fails, then you're going to go through this alerting process. You're going to send an alert. Generally, I like to use Teams, people that are in that Microsoft environment because Teams, people get on their phone, they get whether when they're in front of their desks. Email, they may or may not get, but Teams message, whoever's on call, they could make sure that they've got those alerts on and they are getting those alerts that says, hey, this specific pipeline failed within the orchestration. We need you to go and check it right now. And super high level, that's what that pattern looks like. Then at the end of it, after each pipeline, it actually goes through and sends the metadata for that pipeline to a table to say, okay, this pipeline completed successfully or it didn't if it did fail. And then we have the metadata tracking as well that goes along with it. So there's a lot of configurations there and probably I've just named 3 or 4 different patterns within a pattern. So I apologize.

You definitely have. And then I'm assuming that actually the log from every run goes to some kind of file-based or log-based storage. So you can always go back and forensically see what SQL was run. Or what code was run. And then you probably aren't versioning the data. You aren't keeping copies of before and after data. Yeah.

Yes, correct.

And yeah, so that again, that orchestration pattern, I can look at that and go, yeah, directed graph. Yeah, that is the orchestration flow, alerting, pushing that out to different channels effectively, depending on the urgency of the problem, metadata or storage of all the run stats, because you're going to need those. I'm assuming you're going to have runtime, number of records loaded. You're going to have all the stuff you're going to use data, pushing the actual logs from the run somewhere else that you're just going to keep as cold in case you have to go in and forensically figure out what the hell happened. You're probably going to generate a run ID so you can see that this set of code ran at this particular time and then you're indexing it. If I think about what we do, we do a lot of those patterns, but we do one thing really differently. We don't use a directed graph. So we don't go in and actually hardcode the flow. What we do is we dynamically generate it. And that's a decision we made right at the beginning, right? Now that is a completely different pattern, but everything else, we push to Slack, we don't push to Teams. We log everything like you do. We have a metadata or a context repository where we can see every run and every ID and every table that was loaded. We get performance stats. So we go, that one's taking too long or costing too much, all those patterns for orchestration. Out of interest on that, just again, back going to technology. ADF has been around, Azure Data Factory has been around for a long time. It's a fairly mature product and it's a mature product that's survived many architectures, but many actual architecture changes within Microsoft. So it is a core component. I would have a bet that it's one of the components that won't disappear. And yet I see lots of people doing Azure deployments using Airflow. Now, in the old day, ADF, I think, used to be a full-fat kind of client, but now it's serverless, isn't it? It's more of a serverless type of behavior. So we get the benefits of that. So why would somebody deploy a technical Airflow pattern for their orchestration in Azure compared to just using ADF?

There's a couple reasons. One, I think that people don't see ADF as an orchestration tool. I think that Airflow is the big name out there. I think a CDO, CTO gets a phone call and it's, hey, let's deploy Airflow. And they're like, oh yeah, that's the thing. We're going to use Airflow. Without realizing that I think so many people just schedule and forget ADF pipelines and don't realize that they could actually do a lot of the orchestration within ADF for one. And two, there, there's definitely some additional complexity that can be involved that Airflow lends itself to. I've run into companies where they don't do everything within their Azure environment, right? Where they have different pieces that are running different code in different places. And Airflow does a great job of bringing that all together and allows you to orchestrate it from one place. Where ADF, maybe you're calling an API to get something to trigger somewhere, or maybe you're having to creatively figure out a way to kick something off in some other system somewhere that doesn't have a native connector within ADF. To that conversation that we had earlier about native connectors and whether you go custom or you go use the native connector, right, where Airflow has a lot of those connectivity pieces built in already. The other thing is that ADF has come a long way, I think, especially like a couple of years, partially, I think, because Microsoft knew that they were going to push it into Fabric. They needed to make it a little bit more robust. They needed to get it to a place where they felt like, okay, this is a great tool for Fabric. And to your point of it's not going to die anytime soon, just like SSIS packages are not going to die anytime soon because—

I hope not.

I'm shocked that they haven't, but they haven't. There's that piece. I also think that I've run into a lot of companies that they picked up Airflow because they're like, we've got Databricks, we've got Azure, we've actually got this small AWS account over here in this corner somewhere, and we needed to orchestrate all of it. And that's easier done with Airflow than it is with ADF.

Does Databricks have an orchestrator? A native one?

Databricks has, I will say that in one of the companies I'm doing consulting with, we've written a custom notebook that does the orchestration. And so it's not a native orchestrator like you would see it in Airflow or even Fivetran or ADF. It's all written with Python and we're kicking off child notebooks within the orchestrator notebook that we're calling it. And then we're running those either in sequence or we're running them in parallel because we have that great kind of concurrency and parallelism that we could use within Spark, but it's very custom and it's very created, right? It's a custom workaround.

All right. And then with that master notebook, the controller of the controllers, are you hard coding the things it's calling in that notebook or is it dynamically looking it up and generating that manifest from somewhere else?

So it's dynamically looking it up. Based on a couple of things. We have a table that says, okay, these are the notebooks that we, that need to be run. And then that way it's not a full SDLC, right? It's not a full lifecycle every time something has to change. We're just adding it to this table that's sitting in a Synapse SQL database. Then it can dynamically just change. If we have a new notebook, then it's a little bit less overhead and we're just dumping it in there.

Yeah. The reason I ask is, is this MEGA pattern. And I haven't got a name for it. I've got one now and I've gone through many names and I've hated every one of them. None of them stuck. So I used to talk about config-driven. We had a table, uh, held config. Everything was driven off that config. So that table you hold of all the things that should run, you're going to have some tagging in there. You're going to have a whole lot of attributes, context, and your Megascript, Mega Notebook's going to go through and say, hey, I've been told to run this. What are the dependencies effectively? It's going to look up that config table. That didn't stick. And then I looked at active metadata. That was a term in the market and that didn't stick. Then I tried semantic engine and that didn't stick. I just saw Bolt Speed said, because they effectively use context-driven ETL and they called it model-driven or something. And I was like, yeah, I hate that. Actually, Juan Secada posted something around semantic layers and ontologies as he does. And then I had a comment and Chris Tabb jumped in as well. And right now I'm stuck on this idea of a context layer. Because effectively, that's what you're doing. You've created this SQL table with context. And we have context for orchestration, we have context for the ETL, we have context for the business glossary. It's all context we use. So I'm playing around with that, right? Context layer is this mega pattern. So that's why I was asking, because I was like, I could just see this idea of store the context, use the context to dynamically generate the thing that needs to run rather than hard coding it into that notebook.

And I'm a big fan of that specific pattern where you're using table-driven architecture to do things for a lot of reasons. I think it's a great pattern for cutting down on new projects that are initiated just because one small piece has changed. I'm a big fan of doing that with business logic, having business logic that is table-driven, because business logic changes so incredibly frequently. And you have a table, this is where you're putting your calculations, where you're putting all your relationships and everything, and you're driving all that business logic from this table-driven architecture, as opposed to hard coding. Hard coding makes me wanna pick up my trash can and puke in it.

Yeah, and again, that pattern of hard coding's hidden. So I remember we were back in Oracle Warehouse Builder, where we used to hard code the graph, the DAG. We would say, this node goes to this node. Alteryx came out, and we're hard coding things. Coalesce comes out, and I'm looking at it going, We're hard coding it. It may be pretty and it may look like you're fast, but you're hard coding that thing and it's not dynamic. Then I remember when we went from Oracle Warehouse Builder to Oracle Data Integrator when they bought, oh, that French company, and that was object orientated. You just created these one-off objects and then it built everything dynamically and nobody understood it because it was just too hard. I'm assuming the two people that wrote the product got it, but what I saw was every consultant we had would just basically hard code ADI packages to make them look like we Warehouse Builder tags, that data flow. So I think that this idea of context-driven, we have context about business logic, we have context about orchestration. If we store that as physical data and then use that to generate or hydrate what we need, it's a much better pattern.

It's so much more maintainable than hard coding it, especially when it comes to something that you inherited, that somebody else inherited, that somebody else inherited. Reading through all that hard-coded logic becomes unbearable sometimes.

Yeah. And then actually we still need a map. So while I'm bagging on data flows and these directed graphs and these nodes and links, is not the ideal way of configuring what we do, surfacing it as a picture really helps. Giving a person a map where they can say, show me the nodes and links, the steps in that orchestration or that steps in that ETL, that has value, but it gets hydrated from the core context. It's not. The actual context itself. And so if we think about that, you talked about your 25 years in the data space. There's a bunch of people I can talk to that talk patterns, whether they know it or not. And when you talk about, this is what a pattern is, think of it as Lego blocks. They just, that's how they think. And then if you think about what we've done today, we've gone through some really high ones down to infinite detail. We've said, hey, if you've got orchestration, now you've got to worry about the pattern of where you store your notifications, which channels you're pushing to, what's orchestrating, how do you build your context engine so that it's dynamic orchestration, not hard-coded. Oh, now we need to log every log. How do we parse those? How do we get alerts? How do we get hints on what failed? How do we schedule it? How do we daisy chain it? There's all these patterns on patterns. And if you come into the data world and you haven't done all those, it must be so overwhelming because where do you start? Oh, I'm going to choose between Azure Data Factory and Airflow. So you come in there. And every decision has an impact, because if I pick Azure Data Factory over Airflow, I've now already made some trade-offs, because Azure Data Factory means I can't, in theory, move my platform to AWS or Google Cloud, which I, in theory, can do with Airflow. I still call bullshit on that, because the cost of change is so high, but in theory, I can. Or, as you said, If I need to punch out to a whole lot of systems outside of Azure, you know that you're pretty much going to go Airflow, but you have to know that's the lever, that's the trade-off decision. And so we don't document these patterns. We don't share these patterns. We share reference architectures for a vendor, but we don't share them for patterns. So it must be hard if you come in without all the experience to actually work all this out.

Yeah. And I think for new people, it's already overwhelming because of all those different that are out there, right? Especially from a data engineering standpoint, the vast number of different tools and platforms and languages, because maybe 80% of the market out there is using Python, but there's another 20% out there that's using Java from a data engineering standpoint. And let me tell you, that is not an easy switch to go from Python to Java. Both are OOP, which is, for those of you that may not know, object-oriented programming, but Java, you need a higher kind of technical aptitude for it than you need for Python. SQL to Python to Java is probably a good building block, but you can't jump straight into Java unless maybe you didn't know anything to begin with. And there's so many tools, and it's one of those things where you don't know what you don't know. So much out there. AT&T, I did have a kind of a repository of patterns that we would use for different situations, and a lot of that was driven by what was whitelisted and what wasn't. So what was allowed by the company and what wasn't. And I think even then, because there were like 80 different patterns in there, even that could become overwhelming for somebody that's, for a junior or a fresher that's stepping into that role and they open up that repository and they're like, oh my gosh, I need to learn all this like yesterday.

And unintended consequences. I remember I came in to rescue a project in New Zealand and they'd gone MapR. So it was back in the Hadoopy days where we had a choice of native Hadoop, Cloudera, MapR. There was one more, wasn't there?

Ah, I can't remember.

And at the time, the organization wanted to bypass all their procurement process. So they got one of the shiny suits, large consulting companies to come in and do a quick market scan for them, which is how they used to bypass the procurement process in government back then. And funny enough, that consulting company picked MapR, which They happened to be the only implementation partner in New Zealand. Fancy that. And when we came in, everything was being hardcoded, which is a problem. We want to move it to this context config pattern, table-driven. And so we needed to do some development and we're like, okay, what's the choice? And the choices back then were Python or Scala. I mean, it's like, okay, how do we make a decision? And we said, the MapR team are all Scala experts.

Yeah.

So that's what we should go with because we really didn't care at the time. And the unintended consequence of that was massive because there was no Scala skills in New Zealand. Everybody was either doing R or Python. And what that meant was we could only use the MapR team. And actually the MapR team didn't quite have as much Scala skills as they said. They had to pass it back to the US. And we ended up with an offshore, like we always knew it was going to be offshore because we knew the MapR team was supposedly domiciled. In Australia, but actually their core skills were actually in the US and we had this massive time zone difference. And that one decision, Python over Scala, without bringing in the context of can we buy skills for that in New Zealand if we had to, had so many unintended consequences that caused major problems with that project. So again, it's hard. I think actually in the new gen AI LLM world, I think we have the ability now to potentially we document these patterns, and especially you talked about ages ago, this idea of step-by-step with screenshots. If we do light context of what the pattern is in some kind of structured format, some kind of templated way, and then we do the step-by-step and we put that into an LLM as context, I think we have the ability to give people who aren't as experienced to ask a question and get something back that may help them get through that whole noise, get the signal from the noise. Because like you said, as soon as we take a mega pattern and we break it down, and then we bring in all the technical implications of every different flavor of technology, we can do that. The challenge is nobody's going to write them. Like, who's going to write that repository for free?

Yeah.

Yeah. No way.

Because it's a massive undertaking to your point. I think that's what people try to do with Terraform too, right? Is they try to productionalize all these little different pieces. And then, like AT&T, we had a system where we checked a bunch of boxes for our project and then hit the button at the end of it and it would go through and grab all the Terraform scripts that it needed in order to create and spin up all the resources that we needed for that kind of larger project that we're working on. But similar kind of idea, right? We have an LLM that has in memory that's been trained on all these different patterns that we have out there. We allow people, especially freshers that are learning, right, to go through and just really just talk to it. If you're using like a ChatGPT or something, you could literally just explain your project verbally to ChatGPT or whatever, and then it spits out, okay, these are the patterns and gives you the PowerPoint presentations or whatever the case may be and shows you, okay, this is what I'm going to be doing. And then actually even just does it right. Give it some configuration pieces and allow it to write a lot of it for you. And then you have to go through and comb through it, but—

Yeah, and then you have to go, what's the consequences of it getting it wrong? And do we get it right ourselves? And as I said, I'm not technical. Nigel does all the engineering stuff, but I have a bunch of patterns for working with teams. I can drop into a team and I can observe for a while and go, okay, that's probably where their biggest gap is right now. And then I'll bring some ways of working back patterns, but even then I forget half the patterns I've done before. I'm like, oh, I'm sure I had something that kind of helped the team solve that problem. And then I've got to go search my repository. Do you find that? If you talk about the AT&T stuff, the patterns that you had, I'm assuming they are now either in your head or you've documented it lightly. So when you go and help a customer, you're like, ah, okay, there's a problem. I have a pattern for that. I'm going to grab that. It's 80% of what I need. And then I'll tweak it for their context. Yes.

And that's exactly it. When I left AT&T, I tried to go through and write down, okay, these are all my patterns that I use pretty frequently. And then you tweak them. Because otherwise it becomes this tribal knowledge. And you get that with a lot of companies too, where you have these patterns that have been used for decades, and there's that one person that knows what that pattern looks like, and then they get hit by a bus tomorrow. That pattern's gone forever. So important, I think, to document these things and write them down and actually physically go through and take the time to document it and write it down. Otherwise, you don't want it just stuck in your head or even in a process somewhere. You want it, you want to document those things.

But do you find that even though you've documented them lightly, you still struggle to find the pattern sometimes yourself that you know you've documented?

Sometimes I think this is the pattern that all programmers fall into is sometimes we're really fantastic at organizing things and sometimes we're doing it on the run and it gets dropped into our temp folder somewhere and we've got those 300 files sitting in that temp folder and who knows where they ended up.

And in your consulting business, if you brought on 5 new consultants to work with you, there's no way that you can actually efficiently share those patterns with them. On day one. It's got to be an incremental thing, which is, oh, it looks like we're struggling with that. Here's something that we've done before that seemed to fix it. You iterate it, you pick it up and change it, make it better and push it back. But I don't see a lot of the, that's an overhead. And so the downside of that as consultants as well is that makes you more efficient. And if you're paid by the hour, it actually costs you money to document it. And then you get paid less for implementing it. But if you go to a value-based or you become fractional, then those patterns are key for that ability to deploy something quicker and faster and easier and still get paid for the value of it is where we should be aiming for people that are experienced and have that kind of pattern repository.

To that point, like I said, I try to organize things and then some, and I do a lot of fractional work, right? And a lot of my consulting is fractional. And so I have some old patterns that I could reuse. and to the point of bringing on a subcontractor or somebody, then I could hand somebody, okay, this is my Azure folder. And so here are my Azure patterns or here are my Databricks patterns. But then I have those stragglers to your point of that they didn't make it into the folder because they're somewhere else on my external drive somewhere. And, and then I'm like, where did I put that? I know I've done it 50 times. And yeah, yeah.

Yeah, and that's why a lot of engineers just write from scratch, because it's actually easier to write the pattern from their head, the way they've done it 10 times before, than it is trying to search and find. And that's when it's your code, it's when it's your pattern. Again, when you're trying to share somebody else's, that knowledge of the pattern and the context and the anti-patterns is really hard. So I think one of the things, if you're an organization and you're looking to hire somebody, and that person is saying they're experienced, then you should actually just ask them to describe 3 patterns. If they can't do that, then they're even not as experienced as they said. All right, just Fabric, you reckon it's, it's there now? Would you recommend to a customer, if they're a Microsoft shop, they should include Fabric along with Databricks and Snowflake to make the triad that they investigate or not?

I think it's starting to be. I think it's finally gotten to the point where It could be. I think it highly depends on what your patterns are, to the point of our whole conversation here. It definitely depends on what your patterns are that you're actively using in your business. And I mentioned earlier that Microsoft is really pushing people into Fabric. They discontinued the big Power BI license that allows people to have this Power BI environment, and they're pushing people into Fabric. You have to do the Fabric option now. They forward data engineers, they discontinued the DP-203, the Azure Data Engineer certificate, and you have to go to the DP-700, which is the Microsoft Fabric Data Engineer certificate. And so Azure has come through and taken the stance of really pushing people in that direction. On the plus side, they've also made it so much more robust than it was when you and I probably initially looked at it a few years back. And so there's definitely some value to having everything all in one place. Especially if you're already using ADF anyways, then you could migrate a lot of that stuff over. It's not a push and play, which would be my feedback back to Microsoft is allow people to be able to just migrate their whole ADF environment into Fabric. That would be the optimal solution there in my perspective.

I think again, if we look at patterns, there is a pattern where consulting companies make a large amount of money migrating from one to the other. and that helps the software company because having partners recommend your product is one of the channels. And I think that those core architectural patterns are important. So my understanding is if you're going to deploy Databricks, you would deploy Unity Catalog.

Yes.

The core, it's not optional anymore. It's just a core component. And then if you're going to deploy Snowflake, I would suggest you would start planning how you can decommission all the third-party products because the pattern I see them doing as an organization is starting to become end-to-end themselves. They're bringing, just like Salesforce is buying, Snowflake are bringing it in. So if you rely on a third-party component, you'll start seeing pressure to start using the Snowflake equivalent over time because in their lifecycle as a company, that's where they're going. That's a pattern we've seen before and we shouldn't be surprised when we see it again.

I know of a company in Finland, that's the direction that they've picked up. I have one of my mentees as a senior data engineer for a company out there and they have Snowflake and they're doing everything within Snowflake and they're writing Python scripts, they're orchestrating those Python scripts within Snowflake, scheduling them in Snowflake and trying to do everything within there. But yeah, absolutely.

If you're buying dbt or using dbt, you shouldn't be surprised when more and more features go behind the paywall because that's the life stage, that's the pattern of where that company's at. Excellent. All right. So if people wanna get ahold of you, where do they find you? Do they find any writing? How can they get in touch and see what you're doing?

Yeah, absolutely. So I have a YouTube channel, it's the Data Engineering Channel. So you could find me there. As well as my website, which is gambilldataengineering. My name is spelled a little bit weird, so it's gambilldataengineering.com or, or just gambilldata.com, both redirect to the same site. And those are the big places. And then obviously on LinkedIn, you could find me, Christopher Gamble, on LinkedIn. There's a lot of Christopher Gambles out there, so you'll see the one that's connected to Gamble Data.

I think that's the other pattern. If you want to try and get into the space and make a name for yourself, change your name. Cause Shane Gibson, the guitarist, he's more famous than me. And then Shane Gibson, the sales trainer, he's way more famous than me. And it's damn, I need to change my name. There was a guy in New Zealand changed his name to Mark Rocket because he's just been in, in, in what's the rocket that looks like a penis? The Amazon guy, Bezos rocket. Yeah.

Oh yeah.

He changed his last name to Rocket. So there you go. That's like dedication.

That's awesome.

Excellent. Hey, look, thank you for coming on the show and describing some of those patterns. That's been pretty cool.

Absolutely, thank you, thank you very much for having me.

I hope everybody has a simply magical day. And that, data magicians, was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own agile data way of working, head over to agiledata.io.