Listen now | Recording of AURA - Chris Gambill & Shane Gibson on 10th July 2026

Hey Chris, welcome to the 2nd Aura session. Hey, thanks for joining me. It's going to be fun for a few months at least until we see how we go. How are you going? What's happening?

Too much. Went to the Databricks Summit, then immediately followed by a family trip, just a cruise to Alaska and driving up the West Coast in the US. Lots of fun, all kinds of exciting things.

How many people were at the Databricks Summit? How many do they have those things these days?

Physically over 30,000 people, which was insane. I keep on saying this, but it was like a rock concert. The keynotes, there's lights strobing and there's a little bit of mist in the room. They definitely try to, you know, get y'all hyped up for sure.

And then how many people on the cruise?

There were probably— there were 2,411 people on the cruise.

So, so you, you're gonna come down with a cold, right? There's no way you've been in that mass of people without catching—

No, I had a little bit of sniffles for sure, and I wasn't sure if it was like all the different climates that I was in or what over the past Couple weeks, but yeah.

Excellent, excellent. Hey, just a shout out to Juho, who's there from Helsinki. Jessica, I see you come on, and Andreas, uh, asking how we going. Nice to be here. Yeah, I'm going hot. I'm in the UK, we're in a heat wave. It's 35 degrees Celsius, proper temperature measurement, not that, uh, Fahrenheit stuff. So, uh, yeah, yeah, end of a Friday and I'm sweltering, but that's okay. I got, I got my mate Chris on. We're gonna talk about some data stuff. So if anybody's got any questions, just put it in the chat. Otherwise, as it said in the promo, we're just going to talk shit about data.

Yeah, that's right. We'll just randomly say things. It'll be fine.

Like we normally do when we catch up, right? We're just recording it for now. So go ahead. Go ahead. No, no, you go, man.

I was just gonna say, uh, like there's so many things going on, like just kind of going back to like the Databricks conference, like Apparently the word of all of the summits and those big conferences was context.

Context.

Ontology and context were the big words because the Fabric conference was the same. The Microsoft Build Snowflake, the big word was context and ontology and Databricks was— they're all going to be the masters of ontology. It was crazy.

Yep, uh, there's lots of, uh, lots of commentary about ontology and the, uh, buzzwashing of it by the data vendors, uh, and, you know, we, we hated it in our domain when vendors took words that we'd spent years defining and working with, and now that they're stealing those terms from the, uh, library sciences, it'll be interesting to see how true to ontologies their stuff really is, or Uh, whether it's not. So, uh, yeah, I mean, what did you— did you feel like it was all— well, it's not even PowerPoint now. What is it? You know, they Google Slides, are they vibe-coded presentation apps now? Was it all PowerPoint-ware, or was it a bit more close to reality?

It was, as far as like all the presentations and stuff go. Is that what you're asking?

Yeah. It—

so It was a lot of vibe-coded presentations.

Right.

It was that they used— you could see during the presentation, like, the, you know, the actual Explorer line was Claude.ai, and that was their presentation, right? So it was, it was a lot of that. Not that they weren't— the people that presented weren't very intelligent about what they were talking about, but definitely AI augmented on their presentation themselves. Ed's here.

Hey, Ed. Yeah, and we've got Yuhua asking a hard question. Uh, he's going to be on at the end of the month so he can pay him back, but he's, uh, he's asking you specifically, which I'm glad about because I don't want to have to answer this one. Uh, explain like a 5-year-old version of Genie Ontology.

Like, I always have the hard time with the like I'm a 5-year-old I'll just explain it like he's a data practitioner. Yeah, because you, you know. Um, okay, so Gene Ontology, for those of you that didn't— haven't gotten a chance to go out and look at, look at it or see the conference virtually or be there, basically Gene Ontology is going to take the repository of all your PDFs and all your Word docs and basically read all of that in to determine what the right calculations, what the right source of your data is based on the question that's being asked. So if a, you know, CEO goes in and says, okay, what was my ARR for the last, you know, 12 months, then it's gonna know, it's gonna go through, basically crawl all those documents and understand what the right tables are and what the right calculations and everything that should be made in order to give an accurate answer in a consistent manner. That's the theory. That's the theory of how it'll work. I personally think that there's a lot more work to do on the business side of that to determine what all those pieces are. That's my thing is that Yes, that solves a problem, but it doesn't solve the problem.

Ed asked the question about describe, explain as a 5-year-old what ontology is. I'm going to bail out of that one. I'm going to say go read Jessica Talisman's content because she works in that domain and she's describing it in ways that I can't. But I will argue that from what I've read, the Databricks Genie stuff, may be a genie, but it ain't a horse's ass of an ontology. I think because I'm doing a lot of work in the space from a data point of view, and I think about it as BI semantic layers or metric layers and context. The way I describe it is if you hear people talking about metrics, joins, definitions of words, and that a last-mile tool, a BI tool, or an AI agent, or your LLM is using it, and it's typically tacked on the end of your final consumption layer of your data. If you run a layered data architecture, which you should, I'm not going to use the colors, then that's a BI semantic layer. We're creating semantic meaning and hints basically for those LLMs to query that data.

Yeah.

If I talk about context, what I'm saying is there's a whole lot of other stuff we need. We need other information for other parts of our data work. If we're going to go and get it to create our ETLs for us, if we're going to get it to create data quality or trust rules. There's a whole lot of data work that we can get the LLMs, the GenAI stuff to do now, and it needs more context. That's really valuable, but it's more than just a BI semantic layer. If I think about Databricks and I think about Unity Catalog and I think about their integrated platform, that's where they're going to go. They're going to go to what I call the control plane. I call it control plane because I think it sits above everything else. Sorry, context plane.

Context plane.

I think of it like a control plane, like a security plane, Control plane has security, auditing, logging, all those kind of things that we do. But the thing that interests me was they— I think vendors like to do big numbers, and they talked about 4.5 million ontology snippets.

Yeah.

Right? Which I kind of read briefly, and I kind of thought it sounded like, well, they've got some examples from industries of terms, and they're going to go use them. And, you know, I've been around this game for a while, and we've done multiple waves of trying to create, uh, cookbooks for industry, right? In fact, actually, I was just watching, um, Mother Duck come out with their recipes now, or their whatever they're called.

And I think I have a few of those books too. Yeah, like Databricks Cookbook and all these. Yes, yes, yeah.

And so what there was like when I was at Oracle, there was like, uh, out-of-the-box cookbooks for receivables. You know, you kind of just drop it in. And what happens is it makes sense, right, that 80% of what we're doing is reusable. We've done it before, right? You go and hit Stripe.

Yeah.

Why do we have to reinvent how we join Stripe? 20% of it's unique, right? You actually go and how people measure a subscription in Stripe and it ain't just off one table. But I've never seen it be commercially viable, right? I've never seen people actually adopt it. So is that your take that the 4.5 million Genie Ontology snippets and I may have got the terms wrong, but I don't think I have— was like, throw at it and there's this big wisdom, knowledge, information, data.

In fact, like, what they showed up on the screen was like a knowledge graph kind of thing that when they were talking about it, right, where it would basically— a question would be asked and it would turn— it was like turning red based on, okay, this was the original origin point of the question, and then it kind of branched out looking at all the different pieces that it needed to look at in order to resolve what the question was. But yeah, it very much the way that you explained it, absolutely.

Yeah, so we've got a question from Andreas, Cortex versus Genie, that'll be an interesting one. And he's also asked us Snowflake versus Databricks, who'll be the winner? And I'm not sure that's a question of who'll be the winner in the conference stake or the data platform stake. What do you think, Cortex versus Genie? I mean, have you played much with Coco?

I've very limited exposure to Snowflake these days because so much of my practice is working with companies that are using Databricks or Fabric. But I have some experience with Cortex. I like Genie better. Genie feels more natural to me. But again, to be fair, 90% of my time is in Databricks as opposed to another platform. So that's my take. I don't know, Shane, if you get more chance to be in both. I know that you're not as maybe as much in Databricks, I think, as I am.

Nah, nah, cut me open, I bleed Google Cloud Platform these days. I mean, we build everything we do on that platform. I've coached teams obviously, and one of the teams I was coaching using Cocoa, Um, so I kind of observe what they're doing rather than doing it. Um, and I kind of like the way it had lots of moving parts that you needed, right? So you could give it, uh, trusted queries, you could give it business questions, you can give it a whole lot of reinforcement, uh, context to help it get more accurate. Um, but yeah, never used it in anger.

I like my favorite newish thing that's in Genie, uh, that you could do is like if you build your own harness or you build your own skill. Like you can load the repo with your skill and actually point Genie to that skill either as an individual skill for you or as a workplace skill for everybody that's in the workplace, which is, I think, is unique to Databricks. I'm not sure that Cortex is doing something like that or not.

Not yet. I don't know, right? But you know, I think we see how valuable skills are in Claude and we're going to see all the vendors you know, copy that pattern or implement that value, depending on how you look at it. It's kind of a race, right? So, um, it's going to be interesting though. Do you reckon, do you reckon skills are going to become valuable? And what I mean, not to use, but to build and publish. Like, do you think these are going to become proprietary skills that get paywalled?

I think that's— so there is a ton of work right now in the open space, you know, in the open space that people are working on both harnesses and skills. And I think that because people are doing it in this open space, that I don't know if people are going to come out with a monetization method of monetizing skills or not. I'm sure that somebody eventually will, will be like, I have this very specific, very like top-of-the-line skill, and if you use it, everything's going to be perfect, right? That's going to be the sales pitch.

One skill to rule them all.

One skill to rule them all. But like, in, in a couple of weeks, I'm doing another podcast. And we're going to talk about like meta harnesses and skills and stuff. I'm super excited about that. But it's going to be with Scott Haynes. And oh my gosh, I forgot her name. She's gonna kill me. To be— oh, and Melanie Warwick, she is the dev rel for Temporal. But, but we're gonna do a live stream on meta harnesses and skills, and I'm super excited about that. But all that to say, like, there's companies out there now that are working on creating some meta harnesses that will do some of that work and they're trying to get to the point where they're monetizing it. There is some work in that area now. I think that eventually we'll get to that point. I don't think that we're there yet.

I think it's interesting. I made a comment, I think today in the Practical Data Community, in the PDC, that I found somebody said, am I using open source or off-the-shelf harnesses? There's a bunch of them out there now. I replied with, no, I'm not. I'm rolling my own. Then I sat there going, I'm like a data engineer building my whole data platform, not because I should, just because I could. But what I found whenever I grab a harness, it doesn't do everything I want.

Yeah.

It applies if I want to manage my email, but then I can't get it to manage metrics or manage writing my ETL or manage testing my data for me. I run a personal harness, a personal OS that I use to try and automate some of my life, and I'm building one for our product, which I use every day. For me, I've broken it down to Agile Data Practitioner, which is the data work bit, and the Information Product Builder at the moment, and they have sub-agents within them. I think we're seeing everybody I talk to, who's in the space now and kind of playing with it, they're all building out their own harnesses. And I know lots of vendors, you know, consulting companies are doing it, make themselves faster. And then obviously the big vendors will want you to use their harness because it should be tailored for the way their products work and make you more efficient. So I, yeah, I don't see how we're going to monetize them, but I see how the people with the best harnesses will get customers to be bound to their platforms more because it'll be too hard to leave.

Yeah, yeah, that, that makes a lot of sense to me.

Uh, we've got a question there from Andreas, so, uh, let's, let's ask the— and excuse me if I ever get anybody's name wrong, which I will, so that's just a given. Uh, otherwise we'll make the guess. I'll make my, my friends say it. Um, so the first question is, should they jump into Databricks? So, you know, you work primarily in Databricks. I had Tim on before and he's done a bit in Snowflake. I'm Google Cloud through and through these days. If you're a consultant, what do you recommend? Do you recommend trying to be across all of them, apart from Fabric, because that just sucks?

Fabric has its place in the right size of business. But yeah, I mean overall. A little bit of background. I did spend like 3 years in Snowflake, like only Snowflake before I even knew anything about Databricks. So my first venture into those platforms was Snowflake and then I migrated to Databricks and kind of fell in love. I think that, and I coach people that their businesses are using both in some cases. I have a couple of directors that I coach and their business uses both. And I think that Snowflake is fantastic from the analytics standpoint, but if you're going to be doing any AI/ML type work, I think that Databricks is the place to be, right? That's kind of where their bread and butter is. That's where they grew up from, is that ML place. And so they have a platform that is very tuned into those types of huge volumes of data, a lot of ML-type work. They've done all this work with Genie and working to perfect it. They have all these Spark Lakeflow declarative pipelines, which the name of that whole thing changes every 6 months because at one point it was DLT. They have that, that really does a fantastic job of a lot of transformations and even data quality rules to where you don't really need like a dbt on top of Databricks. I think my preference is Databricks for those workloads, and then you could even still have Snowflake for some of your other workloads. I do also see I have one client that has Fabric for their analytics side of things, and then they went into Databricks for some of their more advanced ML stuff to create their data lake. The great thing about that is that Fabric and Databricks have now recently have the OneLake zero copy between the two, and so that's nice.

When you say they use Fabric for analytics, you mean they use Power BI on top of Databricks?

Right.

I just wanted to check.

Well, not only that, I like they're loading data into OneLake, like they're loading their business data into OneLake and then they're loading all of their engineering workloads and CAD drawings and stuff into Databricks and then vectorizing it from there and going that route.

Cool. I can answer it a different way. I'm assuming it's a consulting question as a consultant rather than for an organization or a person in an organization looking for their next job. I look at it from the 5 E's. If what you're getting paid for is effort, uh, then I'd probably pick a platform and become good at doing effort, becoming visible, right? So that your network knows if you need, uh, somebody to do Databricks, you know, it's Chris. If you need somebody to do GCP, it's Shane. If you need somebody for Snowflake, it's somebody else, right? So you're going to say, if you want somebody to come in and do some effort, do some hours, uh, these are the platforms that I know really, really well. From there, I'd say if you're going to sell expertise, then you probably want to pick one core one and a little edge ones, right? I'd always say you probably want to pick some of the new stuff coming out that's quite small, um, to keep your hand in and give you a little bit of a flavor. So, you know, I'm an expert in Databricks, but I also know a little bit about, um, you know, about these other things so that you are slightly different from every other Databricks expert, right?

Yeah.

But the key is you are an expert in Databricks. You probably are certified. You live and breathe that and that kind of stuff. If you're paid for experience, then I believe you need to suck out the patterns of the platforms and be good at least at 2 or 3 of them. So you've got to understand how all the moving parts, especially with, you know, the AI world. So where's this, you know, the context live? How are they building agents, how they're doing the transformations, all those kind of things, how storage work. And then you should pretty much jump across the platforms over time. Yeah, 'cause you wanna say is, I've got the experience, I've got the patterns, I can work on anything. If you're paid for efficiency, then again, you go back to a single platform, right? And you are focused on reducing costs on that platform or doing the data work faster and cheaper than anybody else. So not hours, but a fixed outcome price. So that's my view. It's like— It depends.

I would agree with that. Like from a W-2 standpoint, like if your focus is to be on a W-2 salary, then I would start with one and then branch out from there because over time you're going to have to bounce from one to the other, right? That's just— that's the reality of W-2 employee. As a consultant, like as a consulting agency, like I focus on Databricks, right? With a side of Fabric. It's how I define it. Because for instance, I'm both, to your point, as a Databricks consultant, I'm a certified professional data engineer. I am a Databricks partner. I'm also a Microsoft partner, partially because there's Azure Databricks and that's That's a thing. So, but, and because of that, I have that side of Fabric. But, but yes, 100%, I agree with everything that you were going over there.

And, and for the non-Americans of us, W-2 means you're an employee. Is that—

Yeah, W-2 is an FTE, right? Full-time. Okay.

All right, excellent. All right, I got a comment or question from Juha, so I'll just read it out. Uh, in the hardest skills discussion, I think we have a bit of an expertise bias, an expert bias Of course, us techies will build our own from scratch. We've always done that for all tech. The average consumer will want to get something off the shelf with very little config or customization. Yeah, I agree. What do you think?

Yeah, 100%. We've always— I like that. Us techies have always— I always refer to it as us nerds, right? Us nerds have always just done it ourselves, right? We're the ones that when we were sitting in front of Windows 95, we went through and basically reprogrammed everything to match all the sounds that we want our sounds to sound like. And like my opening and closing of my computer was like, was a Bon Jovi song back in '95. And so, you know, I did all that, right? So, but yeah, to that point, we'll make our own thing. And then yeah, I could see that where from a monetization standpoint, there would be use cases outside of the techies for sure.

Yeah, yeah.

I mean, I think finance would probably be a really great one Finance, healthcare.

Um, yeah, I think, um, we're in the— we're in the beginning stages of the technology and the wave, right? I mean, there's a whole lot of unknowns. So, you know, everybody who's a tech head in the space right now are probably grokking a Mac Mini or 2 on their desktop, you know, to actually play around with it, or, you know, some Raspberry Pis or some little, little things to kind of isolate, you know, and sandbox what they're doing and see what happens. Um, it's going to take a while, I think, till we get to mainstream. And the reason I say that is if you look under the covers, if you try and do this stuff, these are so many moving parts. There's so many many-to-many relationships, and every week something changes, you know. So in Claude, I'd built my own review process, so whenever I did something, I then sent it off to do a review. Um, but then a whole lot of new native Claude skills came out And so I just replaced it with that, and I'm getting a much better response now. And that was effectively free, right? But I had to know to switch it. I had to know it changed. You know, there's, there's one that I've just been playing with at the moment, which is when you ask it to do something, if it's something kind of important, you're not just chatting with it, you, you tell it to spawn an agent to do the task. And then you tell it after that, spawn another agent to review the task. And that 2-shot behavior tends to give you a better outcome. I mean, you burn more tokens, kill more butterflies, but the quality of the work coming back is better. So I think the technology is changing, the patterns are changing, all the moving parts are changing on such a regular basis that, you know, it might be a while before we get to consumer grade, but who knows.

Well, and I do want to make a comment about what you said there too. Because yes, like the sub-agent concept of spawning sub-agents and having like an orchestrator agent basically that's coaching each of those agents on, okay, and pushing back feedback to each of them as they report back is a fantastic pattern. From a quality standpoint, you get more work. I wouldn't go out and use Fabric 5 because you'll use up all your consumption credits in about 2 minutes. But also, like, we have Omnigent now, and if you haven't used Omnigent, it's really cool because you can wire in your Claude, you could wire in Codex, you could wire in, um, pretty much any other agent and basically have them debate on what the right path is and then spin up sub-agents to actually do the work. And like, you're using all of the LLMs at once. And I've figured out that when Omnigent realizes that you're out of tokens with one, it will take the remaining workload and move it to the other one automatically. That's really cool.

I'm going to go point my harness at that when I've got some more Fable. They reset the Fable credits last night, so it's like, yeah, I was done. I had the red line. Now I'm back into it. So I'm going to maybe point it at that new harness and see if it does it. Yeah, the cost thing is going to be interesting, right? Uh, I mean, the other one that I read the other day, and I haven't tried it, is, you know, let's say you're in Claude Code, uh, you basically get it to write what you want, then you open up another session and you say to Claude that ChatGPT wrote this code, could it review it? And somehow it changes its mindset.

Yeah.

Not that it has a mind, right? So, um, I think it's about removing the memory as much as you can when the review cycle comes in. So it's not just patting itself on the back going, yeah, I wrote that, we're all good.

Yeah, exactly. Yeah, that again, that's why I like playing with OmniGen, because it will automatically just take that context. And it specifically has a debate— what are they called? function basically that you could tell it to have them debate, which is really cool.

Oh, like a red-blue, red-blue team and then a judge? Yeah. Do you find that, um, you now have multiple sessions running at any one time and you're context switching as a human? Yeah, yeah.

I, I would hate to see some— for somebody to look at my desktop literally right now because I have Claude open, I have Codex open, I have Omnigent open. and I have Gemini open and I have Databricks open. It's just bouncing back and forth between them, figuring out what the best answer is that I could get.

Do you find that those tools make you more efficient or effective?

I think it depends. Sometimes it's a distraction. Sometimes I think it's a distraction. I also have come into teams where they were so heavily into AI. And I think that we're starting— I'm starting to see companies kind of backpedal on this, but they were so heavily into AI that they were doing everything in AI, even the simple stuff. But the simple stuff you could do quicker, right, than AI will think about it and return an answer, especially if you have people that are just defaulting to like medium or high for everything. And, and it's going to think and think and think until it gives an answer. So I think sometimes it's a distraction. I think for other times, like there's been times when I've asked a question and it gets me. And I've said, we've talked about this before, it gets you like 80, 90% of the way there. and you're just tweaking some stuff after that. In those cases, especially for the bigger, more complex things, I think that it— I know that it speeds up my work. In addition, like you, like I have a harness that I use for some of my like architecture audits and I just point the architecture to that, to the LLM, and basically it cranks out and evaluates it for me. And I still have to go through and like pick through the responses and just kind of still evaluate the platform. But from a wording standpoint, from a kind of this is the biggest bang for your buck standpoint, it does a pretty good job.

Yeah, I think we're going to see a lot more routing agents come through where, you know, you ask a question and the first thing it's going to do is figure out where the cheapest— which model, you know, whether it's a vendor model or a model within a vendor is the cheapest to get you there. I mean, lots of people are talking at the moment about running the heavy, especially Fable, running the heavy models for planning and then running the cheaper models for execution. But, you know, the thing I find is I'm vibe coding and I'm vibe harnessing. And what I mean by that is I don't have an eval framework, right? Actually, I've got Fable running in the background right now, burning some tokens, building me one to see what would happen. But I make changes and I go, that probably helped me, uh, yeah, gave me a better response in the future, or it doesn't, uh, versus, yes, if you read some of the Anthropic papers, they're actually measuring— well, they say they are— they're measuring, uh, those changes. So, uh, yeah, we're in the early days for that one.

We've got a few questions, so—

All right, you, you take the next 2, which I think—

Yeah, Andreas, uh, kind of going back to Andreas, he was asking if Besides Databricks, what other tools do you guys rely on for ETL process or ELT processes? I guess I'll go first. Primarily I'm in Databricks. Again, that's the focus of what I do. But obviously I've done some— some companies have ADF, Right? And I've been in a couple of orgs where they were using Informatica. Yeah, that's—

I thought you were going to say SSIS there.

Yeah, I'd much rather SSIS than like PowerCenter. So there are those. There are those. There's definitely like usually the companies that are on ADF also still have like SSIS packages out there that they're still working to migrate, but they're terrified to touch because somebody 15 years ago created that SSIS package and they don't want to touch it. It's usually running some finance end-of-month process too that they're reliant on.

Yeah, we effectively built our own ETL tool on our platform. That's what I use every day. Yeah, I think if you looked at the market, I think dbt is still the number one transformation tool out there at the moment. I think with what Snowflake are doing and forking them in and with the changes of them being bought by Fivetran, when the Fivetran tax turns up, I think that will change.

And did you see that they changed their pricing model for dbt?

Oh, there's a surprise. Sorry, that's the point. I haven't caught up on the Fivetran tax. We knew that was coming.

Yeah, we knew that was coming when Fivetran bought them.

Do you know what the change is now?

I know that it's— no, I've not. I saw an article come out and I know that it increased the price structure, but I don't remember exactly.

All right, we need to write a question for ourselves so we can research that for next month and get in there. So I think what we're seeing is, I think we're seeing the core platforms, you know, GCP's got Dataform, Databricks, what have they got for the actual transformation Ecodi stuff? Is that the DLT stuff, isn't it?

DLT mostly, yeah. The expectations within DLT.

Yeah, and then Snowflake forked dbt, but they also bought SQLMesh, didn't they?

Yep.

Yep, so they're kind of blending a few of those. I'm assuming Fabric's still ADF, right? They haven't rebranded something else and okay, so that's kind of where they're still sitting. And then we've got the—

Oh yeah, I was just going to say they have Synapse, but it's really ADF within Synapse, right?

Ah, Synapse is a database, right?

Right, it's ADF within Fabric.

Yeah. Then we've got, and these are all my opinions, but I have them. We've got the legacy Oracle Data Integrator, whatever, ODI is still around. Informatica, still lots of legacy stuff around. There's still lots of ETL tools that have been around for a long time. Then we had the wave before the modern data stack wave, so the Warescapes, Um, they'll hate me saying it, but who's the guys that were in the UK that do ETL and have cloudified it now? I can't remember what they were called. They're still around. Um, and then we had like the new breed come out, um, you know, to try and— they were kind of either right at the end of the modern data stack or straight after it, trying to, you know, take it. Because effectively what happens is it's like a cake, Right? Everybody, they kind of go down, don't they? So you start up at the top being the cool new thing, right? And then you get a bit bigger and you go down to the next layer and you get a few more customers and you move into— you go away from being $10,000 a month, a year, to, you know, half a million enterprise. And then you go down a bit bigger and then you get bought, right, by somebody, and you sit on the, on the bottom of the layer cake and go moldy. Um, so—

There's also like Dagster and Fivetran and like Databricks has Lakeflow Connect from an ELT standpoint, right? Just getting stuff out of source into— but yeah.

Yeah, yep. All right, let's jump over to Mitchell. We'll come back, Andreas, to the other couple questions, but I'll just jump to Mitchell. So have you guys heard of AI cleanup consultants? Have you been approached for that in the context of data engineering projects? And are you familiar with demand? Hmm, I wonder if that— like, you're thinking he's thinking somebody's vibecoded some shit and now you've got to go in and make it run or make it accurate.

Make it secure and accurate. Like, I always had in the back of my head that eventually, like, there would be this concept of we need somebody to come in and clean up what we've done because we've made a mess of it.

AI janitor.

But, but I've not, I've not felt any of that demand at this point.

But if you're going to do it, would you use a GenAI tool to clean up the GenAI tool?

I don't know, like, like it, it would be— I think that would be one of those things where you've built a very custom harness that explicitly does that, right? But But man, yeah, it's like the snake eating itself, right?

Yeah, but if you're going to build it, would you use it to scan the vibe-coded crap code and reverse engineer it into something useful? Or would you take the intent of that code and its outcome, point it at the beginning, and get it to rebuild again from scratch, ignoring the code in the middle? Like, which way would you go?

I like the former. I, I think I would give it the initial kind of use case and expected outcomes. Have it read the code, figure out what it's done and what the gaps were between that and the original vision, and then basically start from scratch. Because I think, I think that fixing it probably takes as much time as actually just building it.

Yeah, see, I wouldn't let it see the messy middle because I think it would influence the models. Yeah, well, not the model, the, the GenAI tooling, right? I think once it sees it, it can't unsee it. I think it'll naturally go in no matter how much you tell it and try and rewrite that spaghetti. So, um, but maybe it won't, I don't know. Something to try. It'd be interesting to see, uh, what kind of markets out there for that. I think there's a market. I'm not sure how you, how you do it. Uh, I'll probably follow the Big 5, right, and all their proof of concepts. Um, that might be the way to go. All right, uh, next question from him. Uh, where are you guys seeing the most demand for ML adjacent data engineers where you'd be happy with ML engineering overlap? What platforms and what tools do you suggest? Oh yeah, machine learning stuff, that's probably— you probably play in that space a lot more than I do.

I mean, I hate to sound like a broken record, but if you're going to be in ML, I think that Databricks is where you need to specialize in right now. I mean, obviously you need to understand the fundamentals, of just data science and machine learning. I think that some of those ML engineers will start to venture into AI engineering. I think that there's a natural progression there for some of that work. Let me look at this question again. Where are you seeing the most demand for ML-adjacent data engineers?

Yeah, maybe, maybe give us a bit more detail about what you mean by ML adjacent. But we saw, you know, a while ago data science become hot, right?

Yeah.

And lots of people going through and learning to do the science of data. And then as soon as they went into an organization, they became data engineers because 80% of the time was getting the data that didn't exist. We see lots of people getting trained as data engineers, but, or, or analysts, but, and becoming analytics engineers, right? And kind of doing the transformationy stuff. Um, what are you seeing though from a machine learning, right? Because in my head it's big FAANG companies and people who are experts in the science of machine learning, and that probably is about 5% of everybody. And now everybody else is just a— not just, but everybody else is just a data practitioner.

Right.

Yeah.

Yeah. And even so, adjacent was a poor choice of words.

Yeah.

Where you feed models. So honestly, so when you go out and look at job postings, you're going to see a lot of those just listed as data engineers. Like you're not going to see ML data engineer, you're not going to see necessarily ML engineer, you're going to see data engineer as the title or senior data engineer or whatever is the title. And then when you look at the description, you're going to be like, Oh, this is a machine learning organization.

Ah, okay, so he's talking about an engineer that feeds a machine learning engineer and they're kind of a posse. Um, yeah, I, I listened to a, uh, presentation by a mate of mine, Sean McGurr, uh, ages ago. Well, no, actually I didn't, that's a lie. Uh, he told me a story, right? Uh, he was in a presentation, that's right, by somebody from one of the FAANG companies. Um, and the guy was talking, you know, about what they did, and Sean He's a Kiwi, so he asked the blunt questions. He said, so how do you get your data? And the guy looks at him and he goes, what do you mean? He's like, well, when you need to feed data for your model, how do you do it? He goes, well, it's simple. I just hop on Slack, do a request, and then a couple of minutes later the data turns up. And he's like, hey, he goes, well, we've got, you know, 5,000— I made that number up, but, you know, 5,000 data engineers for machine learning, and all they do is sit there waiting for our requests and they munch the data and give it to us. I don't think that's normal in most organizations. And I'm with you. I think most organizations, unless you're one of the big FAANGs, is hybrid, hybrid skills, hybrid roles. We're seeing a lot more of that. So that'd be my answer.

Yeah, yeah, I would agree.

Cool. Excellent. All right, let's pop back up. So, and I said, he does a lot of working with teams in Latin America who are looking to overhaul their data infrastructure, common theme. Uh, is the team's eager to adopt modern data technologies, um, and they need the right roadmap to bridge the gap. What do you suggest to connect with them and help them adopt the best use cases? Oh, so this is back, you know, even in the past they would have just bought the 25 most popular modern data stack tools, uh, and then spent half their time integrating it, uh, hoping that, you know, the vendors didn't go under, uh, and then doing the work. What would you do now though? Like, I think you'd go platform now. Like, you just pick a base platform and then fill the gaps, wouldn't you?

Yeah, yeah. In fact, like, there's so much— I mean, think about it, Snowflake, Snowflake or Databricks, now you could just go out and write Python scripts, right? The reality is that you don't need this data tool sprawl that you see, that we used to see all the time, of 25 different tools to get everything done. Instead, you're hiring for the skill set, right? You're hiring human people or AI-augmented people that will do the actual programming that goes along with it instead of hiring people that understand how to use like low-code, no-code tools that are piece-parted together because like that's where CData started, right? So a lot of CData started because they made these custom connectors to all these different platforms that natively SSIS or other tools weren't able to connect to. But that's my thing is pick a tool, pick a platform, whether it's Snowflake or Databricks or Fabric or GCP and learn it and go with it. As far as not knowing what the right roadmap is to bridge that gap, That's part of what I do. I have a potential engagement here next month where they're looking to upskill their team from Snowflake into Databricks and they need somebody to come in and do some training and bridge that gap between where they're at and where they need to be for Databricks. But that is taking time to evaluate the current team, taking time to evaluate exactly where they want to be within that Databricks stack and then creating a custom roadmap for that team to get them there. That way you're trimming out the fat that they don't need because there's a lot of overlap in concept between the two and you're actually targeting and doing targeting training for that group. That's mine.

Yeah, I think when I used to do data blueprints when I was consulting doing that, I think about it as a bunch of Lego blocks and what I'd, you know, I'd have the Lego blocks that normally go together, you know, the capabilities you need. And then I would do the work to keep myself up to date with the interoperability of the Lego blocks. You know, what if you're kind of on this, then you're going to use these ones because they're more natively integrated than those ones. But then it would all be about the organization. You know, the stack or the roadmap or the blueprint for an organization with a 3-person data team versus one with 5, 15-person data teams is very, very different. So I think you've got to fit. And then my view now is, because we know that Databricks and Snowflake and those platforms, right, like they, they won't survive in the form they are right now. Yeah, nobody ever does, right?

Nobody.

They're either going to get bigger, they— or they're going to get bought, or they're going to die, which is probably unlikely now. Um, but what I would now start focusing on is holding the context of all my work outside of the tooling that I use. So I would find ways of storing the business logic, um, completely separately and then using it to hydrate whatever tool I'm using. Um, so I store it over here and then I write really smart stuff that then hydrates into that tool. Now you're not going to be able to natively go from one ETL tool to another in the drop of a hat, But the trans— the ability to move between them now, especially with the LLMs helping us, is, is a lot faster than it used to be. So for me, it's that, that context is the key thing that I would keep separate from anything that I use these days.

And it— I want to talk to real quickly that transition, right? So I have these examples because I do coaching. So I have a guy that I coach in Poland, and his company migrated from Snowflake to Databricks. The cool thing about that today that we didn't have 5, 10 years ago is that a lot of their workload in Snowflake was written in Python anyways. It was already in Git, it was already version controlled. They had to revise their scripting a bit to fit Databricks, but that was a relatively lighter lift than it would have been 10 years ago, right? That's the nice thing about being able to do that scripting in those systems and, and having Git behind all of it, that we just— there was a time in our world that we didn't have all that.

Yeah, so keep, keep that context separate, keep it version controlled, and then, you know, it's effectively a language change. It's still not free. I still see organizations changing platforms for the wrong reasons.

Yeah. Yeah. And thinking it's going to solve all their problems without— and the reality is that there are some bad patterns, there are some anti-patterns that they're using now that are really driving up their cloud costs. And then they migrate to this new tool because there was some promise that it was going to reduce their overall cloud costs. And then as things continue to scale, they still see the same problem because they did just lift and shift into that new tool.

But I'll probably save their costs in the first year, right? Because they got the sniff, sniff, snort offering from the vendor, right? You know, free credits or free compute for the first year.

First hit's free, right?

Yeah, Google do that, don't they? They give you the first year free so you can really get it IV pumped into, into the arm. And the problem with that is it just causes anti-patterns because people don't need to care about the costs, right? It's just lazy. Like, I put my hand up, I'm Opus 4.8 max all the way, baby, until I get close to running out of credit, out of, uh, token limits. and then I, then I care. Um, so, you know, you open up the IV, you gotta, you gotta take it all. Um, there's a question here from Andreas. Uh, any good advice, uh, around the hiring process for an aspiring data engineer, um, and projects and that? Well, my advice is, uh, go and get Chris to mentor you. He's bloody good at it. Um, but that's how you make your living, Chris. So, you know, if you feel up to it, give us some freebies.

Yeah, no, I mean, I am. I'll tell you that, like, I do it looking at the time. So I'll tell you a little bit of a story and then I'm going to get into the answer there. But at first I did free mentoring and I found the first like 3 people that I mentored, like there was no cost. I was just doing it because I wanted to do it and wanted to give back. And it was, it was awesome, right? it felt good. But about 3 weeks in with anybody, I would find that there was no follow-through by the person that I was mentoring, right? And my wife, who is a mental health practitioner, she has her own, her own practice where she does mental health therapy. But she was like, it's because they don't have any skin in the game, right? And so I found that when I start charging people, that's, that's why I charge people as much as anything else, because One, it's part of how I make my living, but now— but 2, people are dedicated when they have skin in the game, for lack of a better term. Okay, so getting into the actual answer to your question. So this is the way I structure my coaching program, right? I have people send me their resume. They identify 5 or 10 jobs that they look at those job postings and they're like, yes, this is exactly the job that I want to get into. I have them specifically identify an industry that they want to target, and I have them do a self-assessment. From there, I take all of that information and I'll go out and research that industry and find like 3 to 5 business-related questions that can be answered with data, right? 4 that are specific to that industry. And then I go out, find some public data sources if I can, that relate to answering that business question, and then I'll build a full end-to-end project around that business question and those datasets. That way, when you're building your portfolio, they're not this generic Titanic or COVID dataset, which for those of you that follow me, you have heard me say, if I see those on a resume, I'm probably just tossing it out. They're specific to the business. When somebody looks at that resume, then they look at that resume and they say, oh, this is an actual issue that we're having. I'm going to go out and actually look at this repo, right? Otherwise, they're— a hiring manager is not going to look at that repo. If they look at it, they see some projects, they're like, cool, okay, they've done some projects and they're going to move on. But if they look at it and see, oh, that's something literally we're working on now, they're going to go and take a look and see what you've done, see what types of decisions you made to do those things. And you're gonna get an interview at that point, right? Unless they go out and look at your repo and you have a bunch of exposed secrets, right, then they're probably gonna call somebody else and have you reported. But, but that's, that's the big thing is build, build like 3 projects that are very specific to the industry that you want to get into, that are very, that are related to the tech stack that you identify in those 5 to 10 job postings. And do the work. Be sure that you document all of your decision points along the way because that's what's going to tell somebody that a real person did it, not just AI. And then when you're putting together your resume, you know, you've got your professional summary, you have your skills, and then you have your projects, right? With the actual, like, impactful pieces of those projects, similarly to how you would actually bullet out your actual work experience. That's my—

Good point there about, because everybody can now just use LLMs to code, right? So that, you know, making the decision points, it looks like it's a human. The other thing I'll add is, yeah, so I mentor university students every now and again, just when they need to do a project. So I finished mentoring one who was doing a master's in data science. And, you know, I got him to do a portfolio, right, to do a couple of projects and have a portfolio website. And the key there was it wasn't about the code, right? It was about the ability to have a clear conversation with a potential employer later. And so what he found was, uh, you know, we gave him— it wasn't Titanic, right? It was things that he was interested in, um, because he didn't really know which industry he was going to go into. So, you know, it was problems that were interesting to him to solve. And he solved them and he wrote blogs about it. And yeah, that was unnatural to him, but that was part of the forcing function is every week he had to write, write the blog. Did 2 things: forced him to do a bit of work each week on the portfolio, but also made him think about it. And then what he found is when he was in interviews and he was asked a question, he had already thought about it to a degree because he had written a blog about it. And so he could come back with clarity of conversation. Oh, well, I did this and then I did this and then I did this. And then he gets nailed with the hard questions, right? But the opening sentence showed that he'd done it before and he knew what he was talking about. And so again, I think, you know, that point you made, it's the doing is as important as the showing.

Yeah. And to that point, like, that's the big thing that I do, like, in my one-on-ones with people is we talk through what those decision points are. Because then you walk into an interview and not only have you thought through it, but when you talk about it, you physically experienced those issues. Like you, you are talking about it the way that you would talk about it as if you've really experienced it in a real work environment. Because you basically did.

Yeah, yeah, I agree. So yeah, do it and then you can talk about it. You know, yeah, go in and get Claude to give you a summary and then, uh, you know, shit.

That's— yeah, yeah, be careful. Excellent.

All right, well, we're pretty much into time. What do you reckon, uh, if you're going to get somebody to do one thing over the next month before we come back? Um, oh, hold on, we've got one more question there. What are things you enjoy and hate when it comes to building and maintaining pipelines? Uh, don't apologize for asking questions, man. It's called Ask Us Anything.

Yeah, this is why I invited people out.

Yeah, we love questions. Uh, okay, so what's the one thing you hate the most when it comes to building and maintaining pipelines?

Maintaining pipelines? The second half of that question? No, it— like, the big— I think the biggest thing that I hate One is because the maintenance piece, you know that things are going to change eventually. You try to build in stopgaps. You try to create your pipelines so that they're resilient against schema drift and you don't have— so that you could continue to have continuity in your business reporting. You put things in place when things change that it still feeds in the data that is critical, but it quarantines the rest. I actually love that piece. Like, so building my building pipelines, the biggest thing I hate about that is when I have to do it in Informatica. For those of you that have not used Informatica, I encourage you to go out and get like a free license to IACS and, and check it out. But But it is— my opinion is that it's a very disjointed platform and it's hard to navigate. But so I love the actual building pipelines when I can actually build the pipeline, when I can actually control a lot of those pieces and build in that resiliency. The thing that I hate about maintaining the pipelines is you can't anticipate everything that's ever going to happen. happen, right? And there's going to be a 3:00 AM pager call for somebody that something blew up in a pipeline that you built, whether it's a week from when you built it or 3 years from when you built it. At some point, something's going to come through that breaks it. The cool thing, for those of you that haven't looked at it, I'm starting to sound like a salesperson for Databricks. But the cool thing is if you haven't gotten a chance to look at what it is yet, the Genie ZeroOps looks like it will be a great tool to help to mitigate some of that work we have to do to maintain the pipelines when things break. But it's not out yet, and in the meantime, we still have to wake up at 3:00 AM.

That was one of those 5-code PowerPoint ones. I think next month I'm just going to say it so I can get it back out of the transcript. is, uh, I really want to go into dead-letter queuing failures of pipelines, right? Like, is that actually a good process, and what are the ups and downsides of it? Um, I think the other thing is, how real are these self-healing agents going to be, right? This idea that there can be a crash, it goes to a Jira ticket, an AI agent picks it up, it fixes the codes and redeploy it. I would really love to know how, how much reality that's going to be. Um, for me, My co-founder is the one that has to deal with the failures, so he basically architects the system so he never gets paged. Um, and I hate writing code, so we've architected the system so I don't have to write code because I can't. Uh, the bit that I struggle with the most is, apart from is what I'm writing valuable, and I'm sorting that— yeah, that one's pretty much sorted now. Uh, what I struggle with is I know it works, But is the answer right? Right. Yeah. So like, I know it loaded, I know that it went all the way through, but how do I know that that number, that metric, actually matches? And this is a problem for us because we normally can't see the source system.

And I would argue that the perfect resolution to that is actually your Information Product Canvas, uh, course that you do on Maven. to do the project gathering and requirements.

So that's good for solving the value problem, right? Am I building a lot of value? But it doesn't solve my— is that number I'm surfacing actually right? Like, I know it's— the code's right, but how do I know that that actually is the number of customers when I can't see the source system that it comes from?

Uh, that one, yeah, that, that I could see where that keeps, keeps you up at night. And The project requirement gathering is, is like a big piece of making sure that that's right down the road. If you don't do that right to begin with, you know the number's wrong.

Oh, actually, Juha's answered it. He's saying make sure you work, you know, if you're the person on call, make sure you work in a time zone that's 12 hours different from—

Nice, I love that one.

AI doesn't sleep, so that's awesome.

Or hire a team that's, you know, halfway across the country. That's You know, the world.

Excellent. All right, well, uh, we're done. That was an hour. Um, call out to anybody watching this, uh, we'll announce the next one, uh, the times for the next ones. Um, they'll be in Substack. If you go into the article, uh, announcement in there, write a question, uh, onto that article. Chris and I will read it, um, we'll go and have a look, um, and then, um, we'll, we'll go from there. Um, yeah, protect— yeah. Checksums are fine that the numbers coming in are correct for me. Like, I know that the data we got is the data we received, but then we do a whole lot of gymnastics to make the data answer the business reality. And how do I know that that definition of, uh, active subscriptions for marketing customers in the Latin South American region is actually the right answer? Because that's not in the source data. So, uh, yeah, maybe a chat for another day. But, uh, yeah, yeah, if you're, if you're watching it, uh, Ask us some questions offline, we'll try and answer them. Ask them live, and as you'll see, you know, you ask them, we'll answer them. Yeah, don't guarantee the quality answers, but I thought they were pretty good. So, uh, yeah, excellent. We'll, uh, we'll see you next month. Sounds good. Thanks, guys. Thanks, everybody. Bye.