Watch now | Recording of AURA - Chris Gambill & Shane Gibson on 11th September 2026

We're live. Hey, Chris, it's been another month. So another Aura session. How you going?

Very good. How about yourself?

I am good. I'm pretty busy at the moment. Like, I'm normally chaotic, but I seem extra chaotic at this moment for some reason. But that's okay. We get there. What's top of mind? What's happening over the last month?

Also a little chaotic. school started, so that didn't help any. But between school starting and I've got a couple of customers that are migrating from like SaaS workloads into Databricks.

What?

Yeah, right. Trying to migrate some old SaaS reporting into Databricks. That's an interesting conundrum. enterprises that are also looking at doing some MDM stuff. I've got a company that wants to do AI with a heap of unstructured data that isn't even started to be processed into a data lake or anything yet. Literally, I would imagine the way that they described it, it's literally like a pile of papers and PDFs sitting in a room somewhere like I could imagine in my head. But we'll see, see how that goes. It'll be interesting.

Yeah, I'm a recovering SAS person. I worked for the company, uh, for 5 years in New Zealand. It was one of the first vendors I worked with. And then, uh, when I started my consulting company back in New Zealand, uh, our primary work was SAS work, right? Because it was so complex, uh, that you really, you know, unless you just had the desktop version, if you had anything server-y Right? Yeah, you had to pay consultants to come and upgrade it and make it work and integrate it. So yeah, so yeah, when I had Optimal BI, pretty much first couple of years, it was nothing but SAS for my sins. I think I even got certified. I think I was even a certified SAS 9x something administrator. Woo-hoo!

I've got a paper here somewhere that I was like a SAS 9 base programmer and a SAS 9 statistical reporting analyst or something like that.

Yeah, but did you write macros?

I did.

Were you writing SAS macros? Yeah, there we go. So you're a proper SAS person. Yeah, yeah.

There wasn't any way you could do it. I mean, you had to spin up a VM and, you know.

Yeah, I remember my favorite PROC was PROC MBREED because it made me giggle. Yeah, so yeah, I'm wondering how they're going. Hey, I mean, when I joined them as a vendor, I moved from Oracle to SAS, and the promise was they were going to list on the stock exchange, right? So I was going to get in early, be an employee. They're going to IPO. You know, I was going to make a bit of money, and then the crash happened, and Jim Goodnight decided not to list. And I heard a rumor they're doing—they're going to list now, right? Because Jim Goodnight's in his eighties, and he owns the company. And yeah, I heard. Somewhere that they were going to go list, but who knows, maybe they are, maybe they're not.

Be interesting, right? Because it's either list it or sell it or kids take it over.

Like, I mean, I think, yeah, I think part of it is that death duty stuff you have in your country, right? Where when you die, the government takes a lot of the money for themselves. So I think the idea of IPOing years ago was to, to help his kids. kind of take the money and not have death duties. But yeah, who knows where they're at now. I mean, it's interesting. I don't— you know, in New Zealand they were really losing market share. You know, Snowflakes and Databricks were pretty much kicking anybody's ass. You know, in New Zealand, R was a big thing because it came out of one of the New Zealand universities. So that open source thing was always where a lot of the universities were training people in New Zealand was, you Originally, when I was working for them, a lot of our work came from government, banks, and insurance, as always, big enterprises. But the onboarding was really interesting. It was because SAS basically gave the licenses for free to universities. And so they'd done a really good job of getting universities in New Zealand to train people on SAS. If you came out of a comp sci degree in New Zealand back then, you didn't come out with Python, you came out with SAS skills.

And so Oh wow.

There was this whole wave of supply of skilled people. And then one of the first, I think the first customer or one of the first, second customer in New Zealand was Statistics New Zealand. And they'd been running SAS, they've been, you know, they run SAS forever. So they started getting really good in-grounds into government. And so it was the supply and demand thing, right? It was really interesting to watch, great business model. But yeah, I don't know how they're going now. When I was there, I watched them lose a lot of market share. I don't know what it's like in your part of the world.

What I do know about SAS today is that there are companies trying to migrate their SAS reports and SAS macros and stuff into Databricks and other tools. It's not one-for-one. It's not like you could just copy and paste all your SAS code into a you know, into a Jupyter Notebook and be done with it. Like, that's not how that works at all.

Yeah, I just called out to Robert Long joining us. Uh, yeah, it's interesting, right? Before I left New Zealand, there was, um, a big government department that was migrating from, from SAS to, uh, what were they going to? Can't remember. Maybe Snowflake. Yeah, they got Snowflake. Um, and what was the ETL tool? Attilian, maybe? I don't know. There was a bunch of— they had a consulting company come in and sell them a bunch of tools. And that project didn't go particularly well. And afterwards, I heard that another vendor had come in with a Microsoft stack and promised them that they'd use LLMs or AI, GenAI, to migrate all the CSS code to basically SQL that will run in Snowflake, uh, run on Microsoft, um, uh, and it would, it would do it all for them, right? Just no, no human involved. And I was like, really? And then I, I actually caught up with some people afterwards a wee while later and said, how'd that go? And they went, yeah, nah, the, the organization got stung for a change request to actually have humans, uh, from the consulting company, uh, do the work, right? So I'd done some of it, And part of the problem is that macro problem, right? So if you look at SAS, if it was just code, right, if it was just SAS PROCs and every program was just, you know, an input, a bunch of code, and then an output, you'd be good. But in New Zealand, at least, every good SAS programmer macroed everything. So it was effectively like a stored process, right? It was like, do some shit, run some stuff over here, and then when you get a response, do some more stuff. So Yeah, how an LLM would be able to— I mean, this was a couple years ago, so, uh, back then it wouldn't have been able to daisy chain through the lineage. But maybe now, right? Like, maybe now it would understand the lineage and what a macro is. And yeah, I'd be intrigued to see how close you can get.

I think it could if— hey, I think there's layers to it, right? Just like there would be layers to that, right? You'd almost have to Give it lines of delineation, right, of where those macros stop and end, and tell it, okay, I wonder if you kind of chunk it out too, right? Chunk the Sass out into here's the macro, go process that, and then tell it, okay, now here's the, here's the rest of it, now do it, right? And if it would figure it out, but I mean, part of the issue too is that those macros were doing a bunch of stuff like SAS was for statistics. Like, I, I don't know that it would one-for-one into most SQL languages, right? It would—

Yeah.

I think you'd have a hard time with it, right? It would be so incredibly complex at parts and so long at parts that I think it would Yeah, maybe be better in Python, right? It just— I feel like at least the macro part in Python, right? Prepare the data and then—

So I think your idea of chunking out, right? I mean, yeah, you know, the standard joke about people going into machine learning or analytics, you know, that 80% of your time is just data munging if you're lucky, right? Um, and so a lot of the CS code I saw was just standard data work, right? It was data transformation. Yes, there was, you know, some heavy statistics and some Enterprise Miner and neural nets, that kind of stuff, but the majority of it was data munging. So I think you could break that out. And one of the first projects we did for Agile Data actually was a data migration. I used to say, right, there's 2 projects I'd never do: data migrations and payroll, right? Because—

Payroll.

Oh yeah, because, because the best that happens, right, is it all works and nothing goes wrong and you get paid and everything else is like a downside. But we ended up doing a data migration. And what we did was we created— I think about it as languages, right? I thought about the source system, the legacy source system is German, and where we're going to is French, right? The new one. And so we created English in the middle. And we, because we had a whole lot of patterns around business process and concepts and terminologies, make— we didn't have an ontology, but we had like a shared language. We ended up migrating the legacy system to our platform using our language of business processes, and then the new vendor told us what data they wanted, and we mapped our processes, our language, to what they needed. And basically, originally we gave them APIs, but then they worked out they didn't know how to query APIs, so we ended up giving them CSVs anyway. But what we found is that really worked, right? Because when the vendor wanted something, they'd tell us and we'd map that to our language. And then when we needed some data from the legacy system, we'd map our language to the tables in the legacy system, right? And that really worked. And I've always been intrigued that when people do migrations, one of the problems is they seem to go from source to target straight away and there's no middle layer, right? No design layer. Then actually I've been doing one with our platform where I've been migrating from a legacy tenancy to a new tenancy for a whole bunch of reasons. But we're actually refactoring all the data work in the middle. Because our platform is so opinionated, I can actually use our LLM harness, PractiQ, to look at the old one and the data, the raw historical data and our rules, and then look at our new one and then look at the new data structures that we're getting from from the source system because it's completely changed and get it to rewrite the rules for me. But that's because it's the same language, right? It knows, it's opinionated what the left and the right looks like. It's using the same concept details, events language, the change rules are the same. So everything's kind of the same. But when we first started out trying to sell our platform as a platform to a data team or a data leader, back in the modern data stack days, One of the major problems we had was we get economies of scale because of the opinionated DataOps stuff we've built, and so we have to hold the data on our platform. That's how we can run a subscription model that is really cost-effective. But we often got from people, well, no, I don't want another data warehouse. Can't you just run on ours? We go like, only if you want us to charge you 100 times what we normally charge. Then we looked and said, well, we could make it free to take their data and all their code and move it on our platform. But that wouldn't work because as soon as we ran their code, we then paid horrendous Google bills because it wasn't optimized or ran the way we did. We always looked and said, well, is there any way we can do some magic? Was it like SQL Gloss or whatever that thing was where we just point at their code and then we come back and it writes our logic for us and their star schema has become our concept details and events. But then what happens with their denormalized tables or their stars and stuff? We kind of got too complex and went, yeah, we're not doing that. But I wonder now, so yeah, what are you doing? If you had to recommend somebody migrating from an old technology and a language to a new technology and a language, maybe refactoring the modeling techniques they use or not, what would you say to somebody? What's the best way of doing it?

I think it's a staged process. When I go in, my first step is the audit, audit and inventory, understand what's there before you even start the solution. That's everybody's first step, step 0. But at that point, there's so many branches. It depends on volume of data. There's so many— I hate to be like that, but there are so many what-ifs. I'm a big proponent of the solution fits the problem. What is the current state? Do you have a good— so many companies that are migrating are migrating to solve a problem. That problem usually is that they have a data swamp now and they think that the new thing is going to fix their data swamp. The reality is that you have to fix your data swamp first and then do the migration. That's the it's hand in hand or you're just lifting and shifting your problem to another probably more expensive platform. But I will say that there are tools that you hear about, like I think it's called Blitzy, that will go through and take COBOL code and translate it into Python. it cut what you think would normally take like 6 or 8 months down to 2 or 3 months in conversion of that old COBOL code. Much to the ear of a couple of my friends that are COBOL programmers.

Money.

There are those tools out there and I think that AI is getting better and better at some of that. Like, but here's the thing is that many people take an AI platform and they, they're stuck with— they stick with one platform, right? Whether it is Cursor, who soon won't have ChatGPT in it anymore, or they're using Codex or they're using Claude Code. But few of them are using a combination thereof. God forbid anybody that's using Copilot.

Bless you all that are out there using Copilot. Not God forbid. I mean, we feel for them. We understand their pain and we look from afar using Codex and Claude Code, and we feel their pain, right? And we hope that at some stage they either get therapy or a product that's a little bit better, right?

Yes, better than nothing, right? So, but yeah, so all that to say, like, one of the best practices would be to make sure that you're not using the same harness and same model to review as you do to code, making sure that there are some best practices there. When you're doing a migration, all that point, do your audit, do your inventory, document all of that thoroughly, document what you want your end state truly to look like. Make sure that you're documenting very clearly what all your standards are, what your expectations are going to be at each layer of your platform, whether you use the medallion architecture or you name them something altogether different. But you're defining what each of those layers actually contain and look like, just like, you know, we had talked about a year ago, right, of having kind of that spreadsheet of each layer and kind of defining out what each one looks like. Take all of that and throw it into an LLM and build a plan, right? And then review the plan, right? Build a plan, review the plan, edit the plan because it's not going to be exactly what you're quite wanting it to be, and then start to iterate through each of those layers, right? Because there's going to be cleanup along the way, making sure that at each stage, both with when you create the plan, I've been using Omnigent a lot lately, both locally on my computer as well as within Databricks. One of my favorite things to do is to just get it to debate what one of the models created, what one of the harnesses created as the plan, and tell me what potentially is really the best option and get them to debate back and forth. Mostly I use Claude and OpenAI, but you could use Pi and Hermes and pretty much anything else that you want to use, even use more than 2. But then you get the best plan and then you could review that. Then because you still have that human in the loop piece, review that and then start working layer by layer. I like to generally take a POC first and work all the way through all the layers with a very specific output model at the end. Whether whatever that star schema medallion architecture, if you're going to go medallion architecture, whatever that gold dataset is at the end and work your way all the way through one specific data model and then wash, rinse, and repeat with each one because nobody should lift and shift the whole thing all at once. That's a sure way to lose trust in your data platform as well as have some major stoppage in data reporting, in business reporting. Anyways, That's my long-winded process.

Dude, there's a lot in there. I agree with the review cycle, right? I'm doing that most times now is that when I create a plan, I will ask it to spawn off other, 'cause I'm typically always in Claude Code. I tend to go across models still at the moment, across vendors, foundationals, like stay within that. But I will tell it to spawn off agents. I'll tell it to direct it to use different models, and then I'll tell the main session to be a judge And that's always interesting to watch what the other models pick up and kind of watch it argue with itself. But it can never work out, right? Because it's one session, it's one foundational. Is it arguing with itself or is it arguing with the agents that it spawned off? Like, you keep wanting to humanize it, right?

I think it's like, I envision it arguing with my daughter. Like, it's, I've spawned a daughter and sometimes it's arguing with, like, arguing with her, right?

Yeah, yeah. So I think like, you know, library session is a thing and it's spawning off other sessions that are other things. Hey, Alex. And, um, yeah, it's— yeah, anyway, um, so I do that a lot and I find I get a better response, a better outcome now. Um, the other thing is, is this idea of, uh, greenfields versus, uh, lift and shift, right?

Uh, yeah.

That's a, that's a decision you've got to make really, really upfront. Yep, because your approaches, your patterns have to be completely different. And often I'll see people wanting to do lift and shift and then they end up greenfielding it, right? Because the cost of converting is so high. When they run some tests, the numbers don't line up even though it's like for like. And then, you know, the question is, if you're like for liking, where's the value, right? Unless you're really on a legacy technology that actually is getting decommissioned. Like the vendor won't support you anymore, or they're, they're kind of caning you, um, on fees. And actually, one of the things I'm hearing from some people lately is that some of the legacy-ish vendors, uh, in the market— and, and, you know, we're talking modern data stack or slightly before— are now ramping up their, uh, annual subscriptions with their current customers to try and retain their revenue stream. So you've got to be, what's the reason? And I don't know about you, but I've often seen the reason for a technology change being new head of data came in and they want to keep everybody busy. So rather than focusing on value, they've replaced the platform.

Yeah.

So I think that's the foundational decision, right? Greenfields versus— versus replump or lift and shift, I think then you're right, you've got to figure out all the standard stuff. What are your architecture layers? And again, Medallion is not an architecture, it's a classification system, but layered architectures, right? Layered data architectures are highly valuable if you believe in them. And so you've got to go and do that work, right? Are you changing that? Are you changing your physical data modeling techniques, right? Are you changing the way you model that data? Are you changing the way you store and create business rules? There's like a whole lot of decisions. Are you taking— changing your last mile tool? Are you going context-first, right? There's a whole lot of core decisions, and they should be going into your LLM planning tool if that's what you're using, right? As— and asking— you should be asking questions. If I do this, what does it mean?

I think in that whole process, what a lot of both consultants and businesses neglect is even the governance piece. When you think about tagging strategy and how you're going to apply just governance in general across those platforms, that all needs to be part of that plan upfront so that as you're putting all that stuff, migrating all those things, because I guarantee you that tagging and governance in your old system probably isn't quite going to work quite the same in your new system. You need to understand what that is and even optimization patterns. To your point of lifting and shifting and all of a sudden somebody's got some big query bill, to your point of just lifting, shifting somebody else's stuff into your platform. getting some huge BigQuery bill. Same thing happens in Databricks and Fabric. You lift and shift into one of those tools and all of a sudden you've got a $10,000 bill at the end of the month that you were expecting $1,000 and you're like, where did this come from? It's because optimization techniques just aren't the same in your old system as it is in your new system. Definitely if you're on-prem right now, If you move from on-prem into any of the modern stacks, optimization is not the same at all.

We only just got surrogate key, unique keys in BigQuery. They only just turned up. We've never needed them, but things you rely on, like your partitioning strategy in BigQuery, you need to partition. Potentially you need to cluster it, you need to understand how that works, otherwise you're getting a full table scan or the one I talked to, The number of times you people see people doing a query with limit on it and then they're surprised they get caned for that query and you go, you do realize it limits the rows returned to your screen, it doesn't limit the rows queried.

Right.

Because it's a BigQuery thing. I think the other thing is foundational pieces. If you're going to do this, put in your monitoring, your cost and ops stuff early. Right? Uh, the second thing, another thing you said is, is thin slice, right? You've got a hot— you're moving so many parts that, yeah, you've got some theories, right? But there's a high level of uncertainty. So have a punt, figure out what you think your plan is, and then do a really thin slice to figure out where you got it wrong and what's going to fail. I think that's really important. Don't, don't go big bang on day one.

Yeah, never go big bang on, on day one. I love the comment about having your your monitoring strategy in place upfront. From cost monitoring to failure monitoring, all of those things should be set up upfront. When you're doing that thin slice, compare that cost between what it was costing you and what it is costing you now. One, that when you're done with your project at the end of the day and you're reporting back, Hey, we finished our migration project, then you could report back, this is really what the ROI is of this whole project, as opposed to being like, we were paying about this, but for the last 6 months we've been doing this migration and we're kind of double dipping. And like, otherwise there's no real recorded way to track what that cost benefit has been. Yeah.

I agree. And yeah, again, I struggle with a lot of the cost-benefit analysis I see for replatforming. It's a vanity project or CV project or whatever. And yeah, I mean, you can't stay on old stuff forever, right? It will hurt. But here's an interesting question, right? Like, you'd never buy a car where you couldn't see the speedo didn't work and you couldn't see how fast you're driving, or the fuel tank light didn't tell you how much petrol you left. had left, right? Like, you just wouldn't, right? It's not fit for purpose. And I haven't, like, I haven't played in Snowflake and Databricks for a— or Fabric for a— well, I've seen PowerPoint, sorry, uh, Databricks and Snowflake for a long, long time. But my, my perc— and Google's no different than this actually, uh, my perception is if you go to those platforms, you still have to go and bake in your own FinOps cost monitoring, your own data quality tests and rules and results, your own logging, your own optimization, all those things. I mean, shouldn't they just be out of the bloody box? If I'm buying it—

Right.

Like when Snowflake was a cloud database, right? And it was a really good cloud database, but it called itself a platform. I'm like, well, there's a whole lot of stuff missing. Then it was a data cloud and it was like, Yeah, but I've got to use a whole lot of third-party tools to do what I actually need to do end-to-end. And what is it now, an AI platform, cloud platform, whatever? But like, where's the— where is it at? Because you're closer to these ones than I am these days. Like, if I was going to stand up one of those, like, to prevent, you know, subscribe to one of them, is it all out of the box now, or am I still building shit to do how fast am I going and how much fuel am I using, right?

So first, Thanks for setting me up like this. This is perfect.

I didn't mean to do that.

That was fantastic, a fantastic setup. 2, right this very second, you're either purchasing something else to layer in on top of it for monitoring and cost observation, whether that is an Orchestra, whether that is a Datadog, Whatever that looks like for you, especially if you are bridging multiple systems still. If you have some on-prem SSIS-type workloads, you still have something in BigQuery and you've got something now in Databricks, something like Orchestra that does orchestrate all of them and gives you a single pane of glass for what that observability looks like across all those platforms is a good tool. But if you are migrating to Databricks, which obviously is my realm, there's a couple of things. There is Genie ZeroOps that is getting ready to be GA before too much longer, which is a platform that will do your monitoring and observability across your platform. Will monitor all your pipelines, will let you know when there's cost spikes for a given pipeline or something, and even give you optimization suggestions for those things. In addition, not optimization suggestions, but if something breaks, it'll give you, go ahead and write the code, even spin up a PR for you to be sitting there and look at and review and approve when you come in the next day as opposed to getting an alarm at 2:00 AM to come in and fix it and look at it, which is nice. I'm super excited about ZeroOps. As far as cost observability, I've personally— I have some out-of-the-box canned reports that I create in customers' workspaces so that they have that cost observability. visibility upfront to where they know by user, by workload, by workspace, by tag what that cost is across their environment and can see it either at a granular hourly level all the way up to by a month level and can see what that usage is both for their actual data engineering workloads as well as for like in Databricks, Genie work usage, which is Databricks' AI. That way you could also see who your big hitters are as far as what those DBU costs are in your workspace. Again, I come in and I set that up. One of the first things that I put in the workspace is that so that when I do start migrating stuff, I could watch that DBU usage. Then they are super excited about ZeroOps coming out super soon. It's in private preview now, maybe public preview soon in the next few days, but it's going to be really a fantastic tool that would be out of the box for monitoring and observability across your workspace.

Yeah, I think the Bauplan team, I know they've been working on a ZeroOps type of self-healing agent that sees a failure in a pipeline and goes in. I think everybody's racing towards that. I mean, I have a problem with that, right? I'm like, do we really trust it? But then I remember once Cicada saying, back when we had, you know, uh, Ask AI has been the hot thing, ask a question, and everybody said it's probabilistic, nobody trusts it. And he said, well, nobody ever reviews the code that an analyst wrote to give a number to the chief executive either, did they? So why do we trust the machine less than we trust the human who's doing the same thing? And then I think about that, zero-op self-healing pipeline stuff. I'm like, well, I worry, like, I think about it, I think it's just a bad idea. And then I go, but how different is it to a data practitioner using an LLM to look at their code and an error log, rewrite the code, submit a PR, and another human coming in or a machine reviewing it and going, okay, right? Like, actually, humans in the loop, but it's not. So intriguing.

I have a team that I'm working with now that there's 2 engineers on that team, and one is incredibly into AI, right? Pretty much using AI for all of their development. And the other one is really AI adverse, like they'll use it for something similar. I call him the Stack Exchange AI user, right? And he's using it for that type of programming and stuff, just copy and pasting. He's really averse to it, like he's that fear of, do we really trust what it's doing? What it's doing is sometimes a little bit over-engineering. AI has tendency to over-engineer, especially from a data perspective. Do we really understand it? When this breaks, are we going to be able to quickly fix it or are we going to rely on the AI to fix it? Then do we become chained to using the LLM to figure out what's going on? That's the same fear that I personally have with the whole overall market right now. of the fact that if you go out to like the job market right now, you see a blue million, like staff, senior engineer positions, data engineer positions, like they are of abundance. But you see little— there's always been little to no junior engineer positions, right? But there's like no junior engineer positions.

Yeah.

And there are, there are not a whole lot like lower mid. engineer positions. And so at some point, we still have to train and educate and provide that foundational knowledge for the— and that those decision-making skills, right, for those people that are coming up. Because at some point, I'm going to retire, right? I mean, at some point, you and I are going to be like, all right, we're done. We're out. It's your all's turn. And they're gonna be like, I Oh, well, ChatGPT will do it for me, and Codex will do it for me, and it'll be fine.

So yeah, and we often, yeah, we often say it's the problem with the fact that those LLMs are probabilistic, right? But actually, it's also the complexity we had. So I had a beautiful one I posted on LinkedIn. So, you know, with my harness, I'm getting it to write my change rules now, right? So writing the transformation code, but using our change rule logic, and It's getting better and better, right? I've just been doing constant reinforcement of our policies, ways of working documents, our guides, our golden path, whereas I'm getting a much better hit rate of it creating them properly, them actually working and being what I want. And then one of the things we're doing is building some products, some information products for some customers, and then they're going to take it to market as commercial product. So we've effectively been creating demo environments. Right? So I've been using GenAI to create simulated data that kind of looks right, but nothing to do with their data, right? And so I need to reapply those change rules from the customer's logic, right, for their product across this demo environment to then boot up an information product that looks like what they see, but completely safe. And it's a great use case, right, for the harness because it's like like, you know, the platforms are opinionated, so it's like that over there, move it over there. I don't— shouldn't have to like bring it up and copy and paste it. Um, and normally it does a really good job, and this time, you know, we run a layered architecture and it's opinionated, so we have to go from history to our design layer, right? And we have to go from our design layer to the consume layer, right? Because the consume layer is automated. Um, and the bloody thing goes and writes me a change rule that goes from history to consume direct.

All right.

And it won't run, right? We have controls in place and it just won't run, it won't validate. And I saw it there and I'm like, why'd you do that? And it went, oh, because, um, there's no rules for the middle tier, for the design layer, in the tenancy you pointed me at. And I'm like, really? And then I went, ah, shit, yeah, we run a data clean room. So for this customer, so actually that logic is sitting in another tenancy for the the provider of the data and then gets filtered and we only share the last mile, the consume tile from that tenancy across to our customer's tenancy, which is what the information product built. It was actually right, but it was wrong.

Yeah.

What I did was go on to change it and say, oh, actually hold on, go look at this other tenancy, grab the change rules from there, put them over here and then we'll be all good. It's like, okay. But if I didn't understand the complexity of what we had under the covers and all those edge cases, I mean, it wouldn't have run, but yeah, I can see that.

It wouldn't have run. And would you have been able to know that that was over there to even go out and get? It would take—

I didn't know because we built it and we know how that works. I've got to now update the guys to say there's a thing called Data Clean Room, the patterns are slightly different. I've got to describe a whole lot more context. That's kind of where I wanted to go with this one, actually, because, you know, one of the things I love about the Aura sessions is we kind of sometimes message each other beforehand going, yeah, what's happening? What should we talk about? You know, we kind of did that a week ago. And then as we're talking, one of us says something and like, that's the whole session, right?

Yeah.

So, um, and so one of the other awesome possums is Anna, and, you know, she's working for a hospital in your country and effectively greenfieldsing, right? So new, uh, electronic health record system. And one of the things she's really focused on right now is capturing that context, right? Because she said, uh, they're all designing new processes on this new system, and actually they've got very expensive medical practitioners in the room doing that design, and the context of what they're talking about is so valuable that she doesn't want to fall into that problem with that once that system's finished, the data people have to go in and actually rediscover that context with these really expensive stakeholders. Like, why don't we capture it? And so if we think about migrations, there's 2 things that I would do if I was building— if I was doing a migration in an organization right now. The first thing I'd do is every conversation around that migration, I would record the transcript and hold the transcript raw.

Yeah.

Yeah, because you're never going to capture that context of why you did these things. Like, why, why did we decide to go dimensional modeling? Why did we decide that that was, uh, a fact? Why did we decide that was a Type 6 dimension, not a Type 2, right? Like, yeah, whatever. Um, and the second thing I'd do is I would re-architect so that I hold the context somewhere safe and I hydrate all code on the platform from that context. Because I know then if I'm going to migrate to a new platform in 3 to 5 years, actually the rules, the logic, the context is held separately from the execution platform. In theory, that would help me migrate quicker and faster in the future, in theory.

It would. I would argue that that pattern in and of itself, I think that that's a pattern in and of itself right there that we should have been doing for a long time, capturing the context of what the business users have been doing. One of the biggest things that I did early on in my career was to actually sit down with the group that I was trying to start doing some report automation for and just sit there and watch them do the process. Because it's one thing for them to share what they do in the process. It's a whole other thing to actually sit behind them and watch them do the process because people leave— people are human and they leave things out. But recording all that context, recording what's going on, recording all those conversations gives you both the context for the migration and gives you all those business rule decisions that you will also need at some point for like your knowledge layer, your ontology layer in your data platform, so that when you do at some point layer in an LLM like a chatbot or something on top of your data, then it has that context for how to make those decisions and what datasets to look at and what things to look at in order to provide the answers to the specific person that's asking. Because when a salesperson asks, hey, what's my revenue? And a marketing person asks, hey, what's my revenue? They may use the same words but mean 2 different things. What are my active customers is another good one. How many active customers do we have last month? Marketing and sales have 2 different numbers because they care about 2 different things. But having all that contextual layer in your data platform, is what makes the difference of an LLM bot that's sitting on top of your data that provides accurate information and trusted information and doesn't, or at least it will when you get to the point.

So what do you reckon would happen if somebody, somebody said to you, you know, come and help us do a migration, right? Uh, and then you went back and said, yep, that's fine, but the first step, uh, let's do a plan. Good. Yep, that makes sense. Good, good. And then the second step is to spend 6 months doing an ontology so that we get the context right before we migrate. Like, right, we know they're going to laugh, right? We know, we know it's the right thing to do, but we won't do it, will we?

No, no, we won't. And like, one of the things in my audit that I do is to sit with whoever the data platform manager is And, and hopefully, if I could get enough attention, one of their like senior engineers, if they have one, right? Sometimes they don't even have that. And understand kind of, okay, I've looked at all this. I see all of it. Tell me the history, right? Tell me what played into how we got here. Tell me what decisions were made, what decisions were thrown out, why these things are the way that they are. Because sometimes things are done a certain way and you look at it and you're like, this is not optimal at all, this is crap. And they come up with this very legitimate reason why it was done that way. And sometimes that, that way was because that's the way we did it 15 years ago and we don't know any other way. But other times there's really really legitimate reason. But yeah, nobody wants to— wants a consulting company to come in and spend 6 months gathering all the context across your business. So hopefully you've got something that we could start pulling from, right? But ultimately, it's a work in progress, right? For most companies, even today, it's a work in progress of— you could use another great use of AI is to build an agent that will go in and actually interview people in an organization and gather that context for you so that you're not taking 6 months, so that they could do it at their own leisure when they have 30 minutes, when they have an hour to sit down and go through that interview. Gather some of that information, and then we could turn around and pretty easily summarize it. Like, AIs do a great job of taking the information, you know, just a stream of thought, right, and summarizing it and giving us kind of the output. Sometimes it ad-libs a little bit, so you have to kind of curate it. But yeah, that's, that's a great use too.

So yeah, again, one of the interesting things I keep going back to, you know, whenever we do data work, we often say, just give us the raw data, right? Don't, don't, don't apply business rules on your side, because then we can't see them. And then when the numbers don't line up, we can't see where the problem is.

Right.

And more and more, I'm going back to give me the raw context, you know. So one of the examples I use is, yeah, we'll prototype an information product, we'll give it to the stakeholder, we look for feedback, and they'll— we'll either have a conversation with them and record the transcript, or, um, you know, they'll send us an email with, yeah, I'm kind of thinking I want to change this, don't like that, have we— can we do this, right? Um, and what I found was I, I was becoming the translator. So again, using my standard language pattern now, you know, they were talking German, I'd convert it to English, then I'd give the English to the LLM that was talking French, right? And what I found was when I was summarizing or I was changing the context, and therefore, um, I wasn't getting as good a result as I could, right? So now what I do is I take the transcript, or I take, uh, their email, right? And then I give that to, to the GenAI tool, and then I then sit in the loop, right? I then watch what it comes back with and then, you know, work with it to get it to where I think it's fit for purpose, and then go back for more feedback. So I kind of get out of the way, right? Right? And so, uh, that, that looks good. But then I think, well, maybe I can go to the next level, right? Why can't I just get them to email my bot, right? And it can do the work. And then I start thinking about what they're paying us for, right?

Mm-hmm.

Uh, like, where do we add the value? It is that human in the loop. And then I think about how, what would, you know, lots of people are scared of losing their jobs to AI for good reasons. And so if we've now got an AI bot just basically draining their brain, right, of everything they know about the organization, their actual value, their subject matter expertise— if their value is a knowledge worker, not a, you know, person in a, you know, doing manual labor to a degree— why would they, right? Why? Or even worse, would they throw in some things things there that are wrong on purpose, right? And then we all blame the LLM. So I just think, yeah, there's kind of a line that we could take it, and I'm not sure we should take it. I mean, what do you think?

I like to believe naively that people are good and, uh, and that what I get from things are mostly true. For the understanding of the person providing the information. I do 100% see those people that are— and that's one of those things, I guess, that if you're not in the room, you don't get the tone and the context necessarily to pick up on. This person's kind of jaded against this process, right? So, but there's a balance between taking 6 months to do the research and getting AI to assist you to do some of that research, or to not do that research and to do like a sample of everything and let the business continue to grow it as they have time to continue to grow it. Same problem is going to happen though. You're going to get people that Provide bad data that's gonna output bad results, right? I mean, that's just human nature, I think, to a point. But there's always somebody out there that's not quite right, as much as I like to believe that people are are good. Yeah, it's an interesting conundrum, like. What— where's that balance between it? And that's why we have to come in. That's part of the larger sample too, I guess, of if you have a bunch of examples and you have an outlier that what they gave doesn't match up with what other people gave, then, then I could go out and re-interview that one person. Right.

Um, personally, how would you see it? I mean, again, I just kind of think about the, the signal loss of the noise, right? It may— maybe not, but I remember 20, 30 years ago running a workshop with a housing, uh, social housing organization in New Zealand. And, um, so we ran— it was back when, you know, I was doing the business event analysis, uh, BEAM workshops, right? So we'd get them in the room and work through their business processes using data. And so they effectively would step us through the states of things, right? So, you know, somebody comes into a housing office and says, hey, you know, I need to get a house, right? And they're like, yep, that's good. The process is cool. You see somebody, they do a quick assessment, they go, yep, right, we'll give you an appointment in a week's time. You go away and you come back in and they do more of an assessment, right? And then, go through this pipeline of work.

Okay.

And so we're in this room and we had regional managers from across the country. And, you know, so they're like the large regions, they said, this is the process, right? And the business process people go, yep, that's exactly how we do it. And there was this one office right up north and they said, that's not how we do it. And I go, what do you do? They go, somebody comes in and tells us that they don't have a house. We go Right through the process in the room with them at that time. And I go, but why, right? And it's like, well, because this person's sleeping in a car, so we need to get them into a house. And they were right, but the thing is, they were a small office, right? They had a small population. And if you compare it to Auckland, which is the biggest city in New Zealand with millions of people, they just couldn't scale it, right? There was no way. Well, there probably was, but The system told them there was no way, right?

Um, the cost benefit of scaling that. Yeah, yeah, there would have been.

Yeah, yeah. And then, you know, then we get back to what's the cost of a person living in the house and what's the economic downside of them not having a safe place to live and not, you know, not be able to get a job because they can't shower. And anyway, we're a whole lot of things around there. Um, but there was this variation, right? And so we knew there was a variation because we were in the room and it became a discussion, an argument, and we can see the variation. Now, if I got those and got the surveys and put them in, would an LLM go, well, there's this outlier, but I don't care? Now, I'd have to ask it for the outlier, but would I even think that there wasn't somebody— well, I'd probably think—

Yeah.

Actually, 30 years ago, I probably assumed they followed the process. Uh, now I know that when somebody tells me there's a process, the first thing I do is look at the data to see whether it follows that process and go, well, what about this step that you never told me about? Oh, that one.

And I go back to our conversation about, you know, training junior engineers to be senior engineers at that point, right? That's— that comes for those types of decision-making, that type of decision-making and just critical thinking comes from your experience, comes from lived experience and lived trauma of living through that argument, that conversation, right? It's a— and so you would know, you would know to prompt the AI to make sure that it gives you all the outliers so that you could evaluate them against the whole, to where a younger person to that point might not think of that, even asking that question or asking to do it, right?

So Yeah, I mean, the interesting thing is I find that LLMs constantly find data outliers that I can't see. Like, it's constantly coming back and going, well, in this period it looks weird, right? And I'm like, go and tell me more. And it's like, oh, it looks like there was a reload done and blah blah blah. And I'm like, I would have never seen that because it's such a small part of the whole ecosystem of data that at that point in time. that somehow the, you know, we didn't have a trust rule in place that picked it up, and, and so we kind of, we didn't see it at the time. And yeah, it's really good at that. But if we think, let's just kind of go back and finish off this migration idea.

Yeah.

So Databricks came out with their whole industry models, right? I think that was since we talked, right? So I think we've talked about their Genie snippets as a way of getting an out-of-the-box ontology, and now they've got these industry models. We've seen industry models time and time again, wave after wave of data waves that have come out and never really been adopted. Um, and I'm not convinced that this one will. But we're also seeing everybody now being forward-deployed engineers, right? Every consulting company is now—

Yeah, yeah.

Which, you know, is the Palantir model. I'm sure somebody had it beforehand. Uh, kind of sounds like a consultant to me, right? Like you go and sit with the customer, understand their problem, and help tailor and configure your product to solve that problem, right? We used to call it consulting. But then I think, like, are we— are those big platforms, right, going to get to a stage where, in theory, they're going to have an out-of-the-box migration, right? Like, I saw Google do it to Teradata in terms of they focused on if you've got Teradata code, they had a migration tool that would take that code and rebuild it to run in BigQuery, right? Because they were going after the Teradata ecosystem as their target. Like, when will Databricks come out and say, hey, if you're running this stuff on Snowflake, we're not just going to migrate it like for like, we're going to refactor your entire organization to our industry model and our ontology all out of the box, and it's going to run? Like, do you reckon they're going to get there? If you can say, right?

Again, I appreciate you setting me up. Databricks does have a tool called—

I'm not sure if I'm setting you up in a good way or a bad way.

Databricks does have this tool called Lake Bridge. We're going to share for a second. It does fast predictable migrations to Databricks where it'll take various migrations. It uses this tool to analyze all the code, all the data platform, everything that's in there, and will convert it into ANSI SQL, convert it into PySpark, set up everything as Databricks SQL using Photon. and all the open formats that Databricks is. But Databricks does have this tool called Lake Bridge that can be used specifically for that purpose. As with anything else, it is supposed to be like-to-like. There is still— you find that there's still some things that are edge cases that it just doesn't quite capture. So you hit— there's still some work to be done, but that's—

So there's 2 use cases, right? So there's 2 use cases, right? You're on a legacy technology that's getting desupported, or you just need to get off it, right? So good use case. Uh, there's the other use case of I need Databricks on my CV, right? Let's ignore that one. Um, and the use case of I'm a data leader, I'm going to keep everybody busy by implementing a platform and adding my value. So, so let's ignore those 2. Sarcasm's getting me today. Um, so I'm on legacy technology, I actually do have to move, right? I want to do. Um, and well, the second one is I'm moving platforms for cost-performance reasons, but that like-to-like migration ain't going to give me that one, right? It's not going to make it cheaper to run, it's not going to make it more performant, or is it?

So it— that's part of the evaluation that Lakebridge does, right, is to When it's translating it, because again, indexing something on an on-prem system, on an on-prem SQL Server, doesn't give you the same optimization benefits that it does in Delta Lake. It's not like for like. It's apples and oranges, in fact. Optimization techniques are different. It's going to suggest things like liquid clustering in Databricks or certain types of partitioning patterns in Databricks when you're migrating those workloads. Now Databricks has a query optimizer, so when it reads SQL, then it's going to perform its own query optimization before it actually executes that SQL. That's just something nested within the manage tables and stuff within Databricks. But there's things like AQE, the query optimizer, liquid clustering, all those things that are unique to Databricks that do get suggested within this particular platform.

How long do you think it is before they then say, not only will we lift and shift your code, but we'll refactor it to an industry model with an out-of-the-box ontology and pre-built data products and actually refactor it not just from a technology execution but from an actual design structure?

Well, I think it's still a bit down the road because the reality is that there's always— this will do a bulk of it, but there's always these edge cases because they always For instance, when Databricks suggests that a company use this, they usually implement either a forward-deployed engineer or they have one of their solutions architects come in or they have another— like Gamble Data is a Databricks partner, so they'll have a partner come in and actually help perform that migration and go through that evaluation with them and help with making sure that the optimizations are correct and accurate because again, human in the loop is still important. I think it's still a ways off, but it is coming. There's something there.

Because you are an active partner, don't answer this one. Consulting company previous, and I partnered with certain vendors, you know, we used to get paid as a consulting company by the vendor to come in and implement part of the solution for the customer for free, right? It was, uh, kind of a way for a platform provider to get the customer on board where the customer didn't pay for it, right? But it kind of came out of the customer's subscription or their money, right? Um, so I know that still goes on some of the platform players, right? So then I sit there and go, well, okay, so Snowflake now come out with their ontology snippets and their industry models, and as soon as we get to an industry model, right, they kind of look the same. And what stops them now just cannibalizing each other with these paid-for deployed engineers, consultants, right, to just start sniffing at each other going, well, if you're running, you know, a banking system on Databricks, system and get system right. Luckily you're using the industry model and their ontology snippets, so come over to Snowflake and we'll run it there and it's a really easy migration actually. We don't have to pay the partner too much to migrate it and then we'll give you, you know, because again I know that some of the platform providers give you years free compute to get you across, right, the old snort snort snort. Like, and we know that especially Databricks and Snowflake love putting out articles about performance, right? There's another big swinging appendage, uh, event in the last couple of weeks apparently, from what I could tell, where one of them did a white paper about something performance or some feature, and LinkedIn went viral with Snowflake influencers or Databricks influencers, um, arguing with each other.

Yeah.

Um, so we know they love to target each other, right? So yeah, What do you reckon? Do you reckon that this context and shared context and shared language and that is going to get us to transportability in a good way or a bad way and minimize migrations?

I, I don't. Like, I think there will always be something that holds that back, whether it be, whether it be money, right? Pure greed. Or just like for instance, there are groups out there that are both Snowflake and Databricks partners because you could be technically both. Or you could be like I'm an Azure, I'm both Microsoft and a Databricks partner. You could do those types of things. For me, it doesn't matter to me whether they go Fabric or Databricks. Like, for me, it's the same from that point of view. From my perspective, it depends on size and all the things. But I don't know, there— I think that each Snowflake and Databricks have a very specific use case that they're really strong in. And I think that that will always be the case, that there's going to be some type of level of differentiation between the two.

And so Iceberg works, right? 'Cause then all the data's in Iceberg and you can switch between the two of them in a heartbeat, right? That's the thing.

Iceberg also does a great job, right? You know, Iceberg on top of S3 does a fantastic job. I mean, Delta and Iceberg now are pretty much one and the same, right? It's with this last update, so. But yeah, that's—

So as always, exciting times, right? I mean, who knows where we're going in a month's time, right? We'll catch up a week before, we'll have some notes on what we think we're going to talk about, and then you'll say something in the opening sentence and I'm like, oh, data migration, oh, that's interesting, oh, actually that reminds me.

That's fantastic. We always do a good job of like something that one of us are doing, you're like, oh, I'm doing that stuff too, let's go with that. So it's It's awesome.

Yeah, still freaks me out. Yeah. That's fantastic. Freaks me out that with the 4 of you, right? Because, you know, talk to one person a week, the synergies, right? You're all in different domains effectively, to a degree, you work on different technologies, you'll have different experiences, you're all— but the same themes are coming through time and time again, which I suppose is just state of the market, right? But it's, yeah, it's a privilege for me to talk to 4 4 experts and then just watch the same patterns across them and go, wow, that— yeah, that's, that's kind of intriguing.

Yeah, it— and it's comforting for— to me and helps me with my own imposter syndrome to know that they are also talking about those things because they are all 3 brilliant people as well.

So, uh, yeah, but the main thing I know is because my mum's visiting from New Zealand with us in the UK and, and she's not talking about AI, LLMs, no AI. migrations, Databricks, Snowflakes, Google Cloud, uh, any of those. No, no, she's definitely not. So, so we are a small subset of humanity that focuses on that, but that's why we do the competition.

My wife and my mom would also be just kind of glazed over if I ever say any of those words.

So yeah. I mean, how can you not love a good partitioning and clustering strategy to get your query times down and your costs down? I mean, who doesn't do that? That's what I say.

I thought that's every night's dinner conversation.

Excellent. All right. Hey, good to catch up again. We will see you in a month and, uh, look forward to what we talk about next.

Sounds great. Thanks, Shane.

All right. Catch you later. See you.