Listen now | Recording of AURA - Tim Frazer & Shane Gibson on 2nd September 2026

Yeah, so what's, what's been happening? What's top of mind? I know what's top of mind for me at the moment, but yeah, what do you got?

I mean, for me it's Duck Labs and that change in the ecosystem that's been really interesting and see how things are developing there. Um, it's interesting, like Snowflake and Databricks kind of releasing numbers around dbt and like how many, like how much it's moving into, into their systems at a pretty breakneck pace. Um, you know, there's a lot of development happening in the Parquet space, um, for data scans, which I was like, Parquet gets upgrades? Like, when is this happening?

Um, isn't Parquet dead, right? It's got Iceberg, right?

It's like, yeah, right, didn't you know it? Um, it's a database and it's just like, yeah. Um, yeah, those are some of the things on top of mind. I mean, obviously AI, AI, AI, because we all got to talk about it. But I would say the other thing that's also top of mind now is how fast all these frontier models are converging, so fast. And also, the cost is spiking so fast. I was experimenting with Mark Freeman's B-tree. I had to shut it down for a minute because my tokens were blowing up in cost. I was like, oh my God, this is, this is like real, real cash is gonna cost me now.

Yeah, it's moving fast, man. Um, yeah, shout out to Jeff Long and Nick Z joined us. So if you've got anything you want to ask or want us to chat about, let us know in the chat. If you don't, we'll just carry on. Um, yeah, it's moving fast and I've been kind of spent the whole month in build mode, right? So As I said on LinkedIn, I've spent bugger all time researching and I've seen these things come out, like the Orchestra getting funding. Congrats to that team. But I've got big questions about where the orchestration live these days.

Oh yeah.

Databricks with their accelerators. I'm like, are we cookie-cuttering organizations again or we're not? Then dbt Labs. Oh my God. Apache Ozzy, the OSI stuff's now actually under Apache. In theory, it can become a standard, but will it? Then Google just came out with BigQuery Graph, which I've been playing around for measurement calculations. It's a quasi-metrics layer and I'm like, oh, what does that actually mean to me and our platform? Yeah, but you tend to keep up a lot more. sometimes, uh, and what's going on in the market. So let's go through some of those because I'm kind of— it's kind of free research for me, right? I don't even need to perplexity it, I just asked Tim on, on the air.

I mean, this month the kids are out of school and, you know, I have a small place, I'm not as, as on it as I would like typically. I know, what are you gonna do? Uh, I do, you know, and I typically do write like a, like weekly, but I took a 2-week pause just like chaos sessions, but I still feel pretty close to everything that's going on right now. Yeah, I mean, a lot of fundraising. I mean, shout out to Orchestra. I'll be honest, I am pretty biased towards them. I have an investor, just to publicly state that so nobody thinks I'm conflating my bias towards them, Hugo and team. I very much strongly believe in their kind of play. I love tools like Airflow, and there's these quasi-open source ones like Kestra. It's like they're all really interesting in different plays. Yeah, I mean, I've been using their platform well over a year and a bit now. And yeah, I'm very pleased with kind of like the velocity these tools are showing. It's also interesting in the play, like orchestration inside of the platforms is starting to show up. Like dbt is kind of starting to lean towards it, but not— it's like one foot in the pool. not in kind of thing. So it's kind of like always been this like super awkward dbt Cloud play, but I'm starting to see it like, like morph into Fivetran essentially, right? Um, and where Fivetran and dbt Cloud, like, I would not be surprised in the next like couple weeks at dbt, like, that you start to see it more or less become like one surface.

Oh, well, that's got to be the goal, right? Um, and, and maybe one billing plan? No, probably not, right?

Yeah, I mean, it's gonna get close, right? Like, and, you know, with Census and Fivetran and all these things, like, they're all kind of like conglomerating one. I mean, I'm a pretty big fan of Fivetran. I think they've done a pretty good product over time. You know, I use competitors like Estuary. You know, they all have pros and cons and changes, right? It's the same thing as like using open source tools like DLT, you know, Python. You and I are old enough, like, have been using like the Singer framework. I actually know one of my friends is one of the original engineers on that, on that standard.

On Singer? Was Singer tech?

Yeah, yeah, he worked, uh, he worked for them.

Um, yeah, we, we started, we kind of started off using Multano, right? Uh, because Singer had been bought, so we kind of thought it was dead. And then it's right at the beginning, I think, was it Multano versus Airflow back then? Yeah, we went down the Multano line because we didn't Well, in hindsight it was okay because we didn't need orchestration. But, you know, kind of probably wasn't the best choice given Miltano kind of died a bit. I mean, I know Aaron's bought it and is reinventing it. So shout out to Aaron here in the UK. And he's a Kiwi, so good on him. But yeah, I don't know. I've always had a thing with open source and things like dbt Labs and dbt, right? I'm like, ah. But, you know, we use Miltano and we still do. We had to rewrite it a little bit because we want to run it serverless, and back then you had to have a persistent server. Yeah. Um, so Nigel did his magic. But no, we, you know, we actually, uh, we just did a whole lot of work on using it to do, uh, data collection off YouTube. Um, and so yeah, it's still like, I don't know how much of the code we run is still a fork of the original Multano or not, but they kind of all come, you know.

Yeah, no, I was gonna say, yeah, I mean, with the AI tooling now with like clear patterns, you're like, ah, you know, before I would always avoid doing this tap target kind of thing or like grabbing data, but you're like, now I have a strong pattern, like, uh, cool.

Well, and actually, if you're vibe coding, I don't even know where the source of the code pattern comes from, right? Is it, you know, if I'm asking it to go and build a collector, um, from scratch, not using our framework, which is any framework, is it using Airflow? Is it using Meltado? Is it using like— where's the actual IP coming from? Um, But the orchestra thing, right, like, I was kind of sitting back and thinking about it, and I'm kind of interested in your view architecturally on this, right? So my thought process went something like this. Orchestras for ages have been DAG glue, right? You know, nodes and links put together, bundled up, and then kind of scheduled, right? And then we got, I think it was Informatica and a whole lot of other ones, we started getting trigger-based ones with a little bit more granularity rather than just big blobs of DAGs. We, in our platform, we do it slightly differently. We hold the context of the relationships, the nodes and links, in our context plane, and then every time new data turns in or we have a scheduled trigger, we'll read that context, we'll create a manifest of the steps, you know, nodes and links, a big DAG effectively, we'll execute it, and then we blow it away. So the schedule is not actually the the pet, it's cattle. The context around the dependencies, right, is what we care about. But that's still very batch-orientated, right? Like, it's a micro-batch, and because it's BigQuery, we can grab that manifest and run a shit ton of a manifest in a very small amount of time on BigQuery and then pay for it. But it's still bundling a boundary around a series of nodes and links, right, for that orchestration pattern. I keep coming back to the way I'm using LLMs and agents now, and I'm like, actually, there's a mesh architecture where I really need to move to. Because if I say that each node is— let's call it an agent for now, right? It's an encapsulation of some context and some execution. And then I can say that each of those agents have a relationship with another agent that has a dependency, and I use data contracts for that. Do I need an orchestrator? Because now what I've got— an orchestrator, uh, I've got a bunch of nodes that are separate that understand who they rely on and who relies on them. They understand when to trigger themselves. They understand what their job is. Like, I don't need a manifest because they can just bloody fire whenever they need to. Chaos to try and understand how they're running and troubleshoot when something goes wrong, right? But that's kind of where my head's at at the moment. So what, what's your view? Orchestration's still a key thing.

Yeah.

A little bit more detail, please.

Yeah. Well, I think it's like, it's, I think you're onto something. I think you're seeing patterns in the industry about like how all these things are kind of like meshing together. I think orchestration, I mean, at the end of the day, like, you know, good old DAG, it is still, it's something you did. It's like under the hood. Is it gonna be a Completely separate tool. I mean, even looking at Orchestra, Orchestra, and like everybody, how they're like morphing and changing. No, I don't think. I don't think so. I think there are actually times where you do want it to be completely separate. Usually those are like when you want really isolated, really specific, high reliable tools, and I think that's like really important. Like get simple orchestration. But I mean, ultimately, at the end of the day, in a vast majority of data work, is like, does this deliver value, right? Do I need to have this completely separate Airflow tool to move all these things, or does Snowflake's orchestration or ingestion patterns in MotherDuck solve my problem, right? Like, that's the majority of the time, it's yes. It's like you're kind of like over-indexing on toys and then not really delivering value. We talk about this all the time really between you and I, we're like, do you need another tool?

Yeah. Does your data platform become the harness for your data with GenAI tooling? Is that something where the bang for buck you're going to get out of them with the way they experiment and the way they do integration, is that where the value is? I was sitting on a webinar the other day and It was an IBM thing. Oh no, it was somebody had an IBM person on there. And they were talking about the value of the IBM platform and it was integration. And yeah, I was in a snarky mood. And yeah, but it was still factual, right? And my view was, you know, decades ago, I was around the time of Oracle, IBM, Informatica. And we, because I worked for one of them, sold the bullshit about all our tools are integrated. And then I watch, you know, Oracle buy PeopleSoft and not integrate it into Oracle Apps and buy Hyperion, was it, and not really, you know, different technology stack cobbled together, not fully integrated. And I'm like, bullshit on the integration story. But now when we look at things like Snowflake and Databricks, and, and I'm going to ignore Fabric, right, um, maybe like, are they integrated or is it still a veneer of integration over the top?

Are they integrated? No, no. I mean, yes and no. Like, it's so funny. Like, I mean, I think this, uh, we had a conversation about that. It's like the sizzle reel problem where you see the YouTube video, this awesome thing on LinkedIn, and then you go to actually use the thing and then you're just like, I mean, I guess like directionally, yeah, it connects. But then you actually get into like the actual integrations or their so-called like open support or open protocol supports. And devil's in the details, and that's what you usually end up spending like vast majority of your time is fixing that stuff, right? Yeah, there are plenty of times that, you know, I've used like Snowflake, I've done a little bit on Databricks, and yeah, I mean, they'll say this and then like you'll look, go down, it's almost like this asterisk and you scroll down and you're like, oh, okay, it's not it's not quite supported and there's no ETA of when it's going to be fully supported, it just says coming soon.

Yeah, we had that with Google Graph because it looked like it helped us build out our semantic BI semantic layer or metrics layer in a more robust way than we had. It was early adopter or it was pre-GA. We jump in there and I start creating Metrics and it only let me create measures. And yeah, I use a language of facts. For me, immutable numbers that are in the source system, you know, a number that's been created, that's facts. For me, a measure is an aggregation, a sum, a count, that kind of stuff. And a metric is where I get a complex formula, this over this or this divided by this or window on this thing divided by a window on this thing, minusing max on this thing, right? Those complex calculations. So yeah, I go in there, pull through a fact. Yeah, I can see it in my tools. I go and do a measure, so I go to a sum. Yep, she works pretty good. I'm impressed. And then I go to do a metric and it goes, eh, nah, nah, nah, buddy, you can't do that now. Now it's coming out very soon as a next feature, but then I automatically go, okay, but now I want some filters, effectively WHERE clauses, dynamic. I want windowing functions. Then I go, well, it's not that it's not integrated because it is, it's just that it doesn't have feature parity of what I actually need right now. How much of that do you reckon is with the big platform plays? It's integrated but just doesn't have feature parity with the best of breed of what you actually need.

I mean, this makes me feel so old saying this. It depends on the business. I know it's such a cop-out, but no, realistically, like, you know, I mean, I would say very rarely do you need the best of breed. And of course, like, you know me and like how hard I have a hard time not choosing the best of breed, like chronically, chronic, chronic, chronically on the edge of choosing platforms that are sometimes maybe a little too too young. But then sometimes I'll meet the founders and I'll be like, yeah, I actually think you guys or women can pull together.

And I think that you—

I think there's enough— one of the circular answers is, yeah, I mean, it just really depends on the business. If you're a certain size and you can take that take that bet and really swing for it, absolutely you should do the best of 3. Contrary, I think you should do the best of 3 and really trying to find the angle to really find the business. Because right now with AI flattening so many processes and things, the money is in the little edges that you're going to get, the 1%, the 2%, the 3%. If everybody's building the world's best Claude bot or Grok bot these days to optimize the business, what is your edge? Right? If you can literally, if everyone can just go on a repo and grab the single best book written on operating procedures, just consider everybody else has that exact same book. So what is your actual edge on all this stuff? Could it be tools? Yeah, I can say it could be tools. Honestly, I think it's just gonna be the ability to make fast decisions, like take an action, next step, next step. So yeah, you know, sometimes I think, you know, you know, best of breed can help. Um, I don't know, what do you think?

Uh, so I think what you got to do is you gotta, you gotta write down a set of core principles, right? Um, and they're this over that. Um, and actually, you know, really interesting example over the last month or so, which is why we're really busy building. So we have a core principle which is for our stuff, that if there's a serverless Google option, we'll do it, right? It's like, we're on the Google platform, that's the only thing we run on, we want to leverage that puppy as much as possible, right? So if there's a serverless thing and it's part of the GCP, we adopt it. And then if there's not, now we get a bit tricky, right? So we had to onboard a whole lot of publishers with social media data, right? We had to start collecting their YouTube, Facebook, all that kind of stuff. But, you know, we've got, uh, 100 to 200 external organizations that we have to go and get the data from. And if we look at YouTube, there's an out-of-the-box Google, uh, data transfer service for YouTube, but the onboarding process for that is horrific, right? Like, just the, the way that that person has to, uh, create their— share their credentials effectively, you know, tokenization, it's just a complete nightmare.

Right.

And so we're like, okay, we can't use that, right? Because even though under the covers it does exactly what we want, the user experience of these publishers just means that they wouldn't onboard, right? Or we'd have to spend hours, if not months, doing it. So then we go, okay, so our first principle can't be met. What do we do? And then our second principle is we'll just lease something, right? So if there's a service out there, we'll just use it. and then we'll deal with that, the challenge of integrating it. So we're like, okay, cool. Well, there's lots of services out there that allow people to effectively sign up and create a token and put the data into that platform and we could suck it out. So that's the path we went down to. And then we had some problems, right? So when we tested it, we could get 2 years worth of data. As we were onboarding the publishers, that vendor decided that they were having performance problems and cost problems, so they limited it to 30 days.

Oops.

And then we can only get the data back via API, right?

They—

no direct database or proxy database type connection, which is all fine. So we're good at that. So, you know, Multano taps on our framework and we're all good until we worked out that actually the cost went through the roof because we had to pay for the API credits. And we have very little visibility of how those credits work, but we blew a year's budget in a month, right? And we're like, hmm, hmm, yeah, we don't want to buy that many. Um, so we ended up going, okay, to our 3rd principle, which is if a leased option doesn't work, we build it, which is what we've just done. Um, so I, I liken that to if I was green— if I was working, I went in and was head of data, head of data platform, whatever. in an organization. Yeah, I'd set those core principles up front. Yeah, are we a best of breed or are we a platform organization? Uh, you know, are we open source or are we supported open source commercial or are we, uh, closed source? And I would place those principles not as immutable policies but as guidance. We are open source over closed source.

Yeah, right.

Um, which means then when, you know, you go Snowflake, you're like, well, that's closed source. Yeah, but I can put Iceberg there, and that gets me closer to the principle of open source over closed source, right? Uh, it doesn't mean I need to put Iceberg and Apache Spark and all that shit on an AWS platform and manage the containers myself.

Yeah.

Because one of my other principles may be, uh, DataOps or, you know, small teams. Like, there's a bunch of other competing principles, and that's the key. So that's what I'd do, and I'd be very clear where on that slider I'm moving, right? When I make a decision, Which one did I pick and why? And then the last one is be willing to change. And that's the hard one for the corporates, right? Is say, hey, we tried that, it didn't work for a whole bunch of reasons, and now we're going to go do something different, but that's okay. So as a 2-person startup, it's okay, right? Still gutted, but it's okay. If I was in a 5,000-person company and my career was based on making that decision, yeah, it probably wasn't okay, and I'd be paying that company a shitload of money for API credits. Yeah.

Yeah.

That's my view.

I mean, it's good. It's where we have these great conversations and we've worked together. It is like a— it's so funny. You have to make trade-offs, right? And I think most of the time when we talk, you're just like, well, just make the trade-offs visible. Why do you train what for what, right? If you want to support open source, just be clear about supporting open source or what the the trade-off is. And like, so where I currently work, yeah, we use Iceberg because I felt that the— it's not young, it's just like it's hard for small data teams to actually operate on. But like a year ago, I was just like, I know, you know, Duck Lake, I know all these things are coming, I see this happening, and there's Delta and it has, I think at the time, more superior piece. But I was like, it's very clear to me there's a signal as conversion on Iceberg. And maybe I'm feeling a little risky on this, but it's not that risky. So let's converge on this open format. And I think it's going to pay off on that bet. Snowflake support for it was not great at first despite all the docs they had, but it's definitely caught up and it's been almost seamless. Now, like a year ago, definitely was not seamless. It was definitely painful.

Yeah.

But yeah, there was that trade-off, right? Like, I still like— and also it's like, is the juice worth the squeeze? If I stand up an open-source analytical piece or like you said, Spark, like I've run Spark, Kafka, like all those toys before and I know kind of like what the maintenance is, but you know, the business today, like, you know, you see that, you know, from employment, like, compared to last year, like, analyst roles, like pure analytical roles are down like 15%, right? So you can see kind of what— I actually don't see it that way. I see it like a lot of engineering roles like condensing, right?

Right.

Like a lot of people being like, I'm just an analyst role. I think that those are shrinking. They should. I think that that's fine because there's a lot of capability now. Like, I think that really good analysts now can just do tons of stuff they never could do before, uh, with AI helping them, you know, get that way in the systems, you know, just complexity.

Yeah, I think that's a good point, is if you're the leader of the data part of that organization, it's your job to make the bets. Yeah, and make them visible why you've made that bet. But you've got to look ahead, right? You've got to look at the market and go, well, I'm I could do this or I could do that. Where am I going to place my chips, right? Because that's what you're doing. You're gambling not on a hunch, on, you know, there's data and everything, but you're saying, do I think this is where the market's going to go? Like, is it safe to make this bet now for the next— I mean, it used to be 5 years, but it's not even 5 years now, right? Like, things are moving so fast. And that's the other thing then, you've got to architect your platforms and your way of working to be— what was it, combustible? No, composable. What was that? Remember those?

Remember those?

What was it, that last— that reverse ETL bullshit? Uh, I mean, the, the patterns that were doing were useful, but the architecture, uh, buzzwashing of it was bullshit. Composable? Combustible? What was it?

Composable, but combustible is probably what it's most become. Um, no, I think, um, yeah, I mean, it's really— yeah, it's really interesting going back to your conversation about platform platform. I think it comes at core principles for the business. One of the things I was talking to another data leader, Ahmed, Ahmed from Activity Schema, actually I was talking to him last week and we were talking about what he sees because obviously he started his own startup, which ultimately Narrator closed down, but he left behind the concept for data modeling for activity and he's working on a new one called Polyform. and a few other things that I found really interesting in what he's talking about. He was talking about some of the clients that he's been working with is the engineering team are going around the data team. They don't even want them involved. They're tired of the white tower, the gating, whatever it is. The engineers are wanting to essentially create their own pipelines and all this other stuff. I mean, it's the tale as old as time, frankly. It's like, oh, this is new. I was like, no, this sounds standard. Then an engineer actually has to do data work and And they're like, oh shit, this is actually hard.

This is—

oh, what's the definition of time? Well, that depends. And it's like, you know, even with Claude, you know, and we've talked about this before, Anthropic's paper on like their massive, you know, data modeling program and analytical program, it's eerily similar to the same things that stood up for decades now. Just faster, I guess.

Um, well, yeah, on that. So I was talking to somebody the other day because, you know, I, I lament, uh, how many people point to the Anthropic paper and say, oh, they do dimensional modeling, that's the answer. And I'm like, yeah, no, I'm not sure it is. I'm not sure this— the paper says they do dimensionally modeling, right? And that's fine because any modeling technique will work. I'm not sure that's where their secret sauce was. But I was talking to somebody anyway, and I don't know whether fact— what they're telling me is factually correct not, because I can't validate it. But what they said was, uh, yes, there is part of that company that works that way, but there's also a large part of that company that doesn't. Uh, and so you do wonder how much architectural buzzwatching of that paper. And then especially like everybody else has come out, right? Spotify have done— everybody else has come out with their how we use GenAI tooling with our data team and that. I mean, I think the core pattern of you can't reverse engineer the context for your organization out of your SQL. You are going to need human in the loop.

Mother duck. So many people have come out and they've straight up confirmed that. The problem is you will feel correct. You will feel correct. It'll have the language, it'll have a true Turing-complete vibe to it, and you'll probably— a huge chunk of people will be like, yeah, that seems good. and they will continue on.

Yeah, so I think, you know, that part of the paper, because I— it's been my experience, right? Um, so that, yeah, that's good. But I think there again, these people take little sound bites and go, that is the answer, right? Um, and I think, yeah, it's clearly that's all that's needed. Um, yeah, they're valuable, but they're not the answer. Uh, I don't think so. Uh, yeah, it It's interesting, and it's the same with Databricks coming out with their accelerator data models, right? Their industry standard ontology and enterprise data models now per industry, they've just announced. I'm going to chat to Chris Gamble about it next week because he's deep in that Databricks space. But we've seen it before, right? And it's never worked, and I don't know why, because it makes sense, right? It makes sense that if you're in an industry that is a relatively standard industry, that 80% of what you do in terms of your core concepts, your business processes, and your relationships are the same as everybody else. And then the 20% is where, you know, the juice is worth the squeeze. How are you different, or how are you going to be different, right?

Yeah.

But we've seen those enterprise data models come out and reference models and industry models come out every wave and they never get traction, and I've never worked out why. Yeah, what's your view?

Um, I mean, I get pitched freaking data engineering, data analytics, AI startup stuff. Oh, I'm not gonna say every day because I don't go on LinkedIn every day, but every day I show up on LinkedIn, there's a new Y Combinator or a new startup out of somewhere that's gonna, you know, do all the data engineering work. It's so funny, they've all converged to just those patterns. I'm like, why? There's so much more out there and sometimes better patterns for solving all this stuff.

Let me ask you in a different way. You work in a company that has subscriptions. Right? Yeah, you kind of call it corporate.

Yeah.

And, and you go, well, you know, that's an industry standard model, right? Customer subscribes to product. And, and we know that because you're not the first one I've worked with where I— like, the first time I went into one of those companies and go, ah, it's subscription, how hard could that be? I'd never say that again. Um, yeah, because every, every software as a service company I've worked with subscription seems to be harder than customer. Um, like, but it's meant to be an industry— why is it not just an industry standard model, right? That just makes sense to me.

It's so funny. I love those things. And it's like, yeah, like on the surface, right? There's so many things on the surface when you look at it. And then you're like getting into the contract. Oh, you have contracts? And it's like, yeah, we have contracts because like last year we sold it like this and these clients renewed on this value of the subscription, but then this year we have a new subscription, but the old customers don't wanna move to the new subscription model. And you're not big enough cuz you're like sub $100 million company, so you can't really force those clients to the new subscription. So now you have a bifurcated definition of contract and subscription. And I almost exclude, I'm not gonna say exclusively work for startups, but I generally bias towards them these days and I've worked in numerous ones. Yeah, like everyone starts to become a snowflake. You, you start to hold the line about what contracts or subscriptions should look like, and inevitably something comes down into the business that makes it different, right?

Uh, it's a trial and contract. Yeah. Uh, you know, you know what the answer is? You just work to credits and nobody can understand how they work, and then it doesn't matter, right? Because it's— there's one standard. It's a credit. We don't know how we calculate it, but that's okay. 100 credits.

I do love that. Just a credit system. That way, literally no one can figure out what you're actually charging. You get 100 credits. Yeah, it's almost like credit— I feel like credit card points. Like, man, somebody was like, oh, you bet I get 100,000 credit card points for signing up to this new, like, program. I'm like, yeah, but what does that actually mean? Because Before, it used to be, you know, 100,000 credits was like 1 cent USD. And I'm like, is it now? Well, it's like, no. And it's like this pyramid system of like configurations. And that's exactly how I feel like using AI tooling right now for credits. I just like, which, which Amex card is gonna work with what? Because like, I got a new Amex card. And for— and it was so funny. It's like, oh, if you go ChatGPT Business, We're gonna give you 300 credits. You know, it's like, oh, that sounds great, 300 credits. Uh, okay, I'll go for it. And that's like, oh yeah, minimum seats, and then it's like credited for over this period of time. I was like, okay, of course. It's like, yeah, it's not just put it on the card and you're like, you're good to go. No, no, no.

Yeah. And then, um, well, it's like, uh, Claude, right? When Fable came out and then Opus 5, and then everybody's going Is it me or is Opus 5 shitting the bed? And I— and eventually I'd heard enough people that were having the same problems I had, and I was like, back to Opus 4.8.

Because—

and now I'm sitting there going, oh, you know, uh, when do I make the call to make my harness agnostic? Like, because at the moment I just— it's all in Claude, right? And I'm like, I know I've architected it and theory in a way that I could move it to other models.

Yeah.

But I haven't tested it. So I know, you know, I've been around long enough to know if I haven't tested it architecturally, it's just a theory. And it's a theory that probably won't survive. And I— when do I take the hit for that puppy, right? And then 5.1 came out and I'm like, is it going to be better? Is it going to be worse? Am I going to blow my, my subscription in half an hour because I got no idea what it's doing? Like, I was doing a conference today and, um, uh, I wanted to show live basically using the information product, uh, app, right? So the open source app that I've got. So, you know, there's a canvas in there, I export it to JSON, I upload it to Claude, I ask Claude a bunch of questions, right? And that was, that was the demo. So ran it all through, that's all good. Um, it's all pretty quick, you know, turned off, uh, thinking mode, all that kind of shit. Um, yeah. Because I had, yeah, I had a certain amount of time and it was like, yeah, yeah, it's coming. Each question is coming back in less than a minute. Of course, when I did it live, it took over 2 to 3 minutes for each question, so that shit can do. Um, but anyway, so I was YOLOing along, I'm all good, already in the morning, presentation in the early afternoon. So I was like, cool, let's get back to work, right? And so I'm building some stuff and I'm— I flipped over to Fable 5.1 for something and forgot to flip back, so every new session was fabling me. And then I was sitting there going, hey, hold on, hold on, how much have I got left in my 5-hour window? Because that 5-hour window is bang smack— my conference is in the middle of it. Um, and I was like, yeah, I'm gonna stop working now because I got no guarantee of when I'm actually going to run out of that. And, you know, embarrassing. So yeah, I think there's a lack of visibility, right?

Yeah, I mean, it's the classic like black box problems we got going on here, right? We just keep on punting them. It's really interesting. What do you feel like, um, you, you use harnesses, like, are you, you know, are you going all in on Pi or Hermes, or you've spun up your own thing?

No, I started off, I'm a bit like you, I started off with a personal, personal OS, um, as a learning curve, right? It was back in the OpenClaw, so I did OpenClaw for about 2 seconds, then, uh, did a little bit of reading, was like, holy shit, no. But I used it as a learning curve, and then For our stuff, I started building it specifically for what we needed. Um, I still have a technique where what I'll do is when I hear of a new open source harness, I'll get Claude to basically look at my harness, look at the open source one, and tell me what's missing, what's useful, and try and get it back. Um, but I'm at the stage now where I'm starting to question how much time am I spending on the harness versus actually doing the work. Like, I know the harness is giving— making me faster to do the work, but actually, if I look at an investment window, you know, net present value of time, uh, I'm wondering now whether I'm over-indexing on the harness itself because it's kind of fun. Like, I did this, Claude did that, it shouldn't have done that, I don't want it to do that again. So let's spend 2 hours getting the harness up to date to stop Claude ever doing that, right? And I'm like, is that— like, where's that value? Um, I mean, for us, the value is I can give out— I want to be able to give our harness to our partners and have them work the same way I do, right? Without the intrinsic knowledge of our platform that I know I have.

Yeah.

That's the end goal, right? It's a scaling, uh, play. And that's the dream, right? Um, but I still worry right now. And then I look and say, well, how many other people are doing that, right? Like, how many data teams in a large organization right now are spending 80% of their time writing harnesses and not actually doing the data work?

Yeah, I mean, that's a good question. I don't know how— I, I, um, I question on like how consistent a lot of people are playing. I think a lot of people are are spinning up their own playbooks in the company. A vast majority of people, they're all standing up their Claude or ilk of their own Markdown files to feel like they're giving themselves the edge and not really sharing it across the company. Then people are really struggling on how to take it into multiplayer mode. I really truly think that it's kind of like the next level. I mean, I think that's why Claude Cowork was starting to take off a little bit more. But I think like, I think, you know, I've, you know, my wife and I are, you know, we have our own and we struggle to turn into a proper multiplayer mode because we have like really different ways of wanting to work and do things.

Mm-hmm.

Right. And so if you're like naturally different, could you imagine what 100, 200, 300, 500, of 1,000 different people are like. And they're all like, they like their, you know, as you call them, pets maybe, to be kind of like tuned to them. And that's where like, I think a lot of people are struggling. Like, you need this consistency, we'll call it context or knowledge, and, you know, the facts of the business or facts as much as you can, and your personal life, your calendars and all this stuff. But then like the nuances can be like, I think they're a lot deeper than people give credit to sometimes, right? That like that 10% can be so wildly different and the outcomes, and you're just like, I know we're speaking the same language, but you are coming— you and I are coming completely different points of view here.

And I— yeah. Yeah, well, Anna Berggren's actually working on a personal iOS for family for her family, right? And she's open sourcing it. So yeah, it'll be interesting to see whether, you know, how much of that you pick up for your family stuff.

Yeah, it's interesting.

I was, you know, I was going, I'm going the same direction.

There's so many people working on it. I would say like the other thing I get pitched on every other day is probably not just personal knowledge graph or personal knowledge bases for companies. Like you can see Stripe came out with their one and how they do it. Obviously there's GrokBot out there and like how they're kind of like heading towards these different patterns. Yeah, I mean, everybody knows that like the multiplayer mode is kind of like where it's at, um, you know, open cloud.

Yeah, the question is then comes back to, do you buy a harness off a platform vendor because it's embedded in their technology stack, it's integrated, right? It knows how to execute the harness, and then you bolt on your context plane. Yeah, the thing that holds your context which is the second part of what a harness needs. Or do you go with a platform player that's got both the context storage and the execution? Do you buy all 3 separately? Do you go, I got platform that runs the code and execution harness that understands almost like a data integration hub? My context sits in in a graph database or in MD files or in a Confluence wiki or whatever. My harness for the policies and procedures and way of working sits over here and whatever, and it knows it's already been written to integrate to run on Databricks or Snowflake or Iceberg with Spark or, heaven forbid, Fabric or IBM or Salesforce— I mean Informatica. it becomes a language, right? Like, would you—

why?

I mean, it's— there's so many moving parts that that decoupling is going to kill you, in my view.

Yeah, I think, um, yeah, I mean, this is awesome. This is awesome decoupling. Um, but yeah, this, like, this complexity is so funny. Like, the complexity that we have to handle. Yeah, I mean, I mean, I have cognitive overload every day, like, and I, and I read and I'm a copious note-taker, although most of my— half my notes are actually on pieces of paper. Yeah, I do wonder, like, how people are gonna, like, digest and work through all this stuff, right? Like you're mentioning Markdown, Graph, like all these— I wouldn't say toys, but like all these things. And ultimately it's just like how much is going to be decided by the system? And ultimately, and the other part is like how much do you want? Like you, you look at like how many people came back to, you know, like I wouldn't say like fled Microsoft Windows, but you know, um, and went to Mac, right? And you know, I'm a huge Linux guy, or was, you know, all my servers are almost always Linux. I almost never see any use Windows Server, like blah, blah, blah. And, you know, and then what's— DHH announced Omarchi, like, and like Toby Luca and a bunch of other names I'm gonna like destroy. And they basically put like $10 million into making Linux amazing. And I was like, man, maybe this will finally happen. Maybe it won't. I'm not sure. And so I just think things are changing, right? Like, I got really comfortable with Mac and like how Mac does. It keeps my life simpler. And so when you're talking about like harnesses and stuff like that, I was like, I mean, we've already demonstrated constantly, like, whatever's easy, wherever the water flows easiest is where people naturally go, right? Even if it's better for them, right? Maybe if like freedom, open source, Linux, whatever, The vast majority of people will be like, okay, if I don't have to think about this for 5 minutes and I have to pay somebody $5 or $20 a month, then, you know, a huge portion of people just like don't care. Although the irony to me though is how many people don't pay for these OpenAI tools or like things and they just are willing to take the ads.

Hmm.

And I, you know, taking income across distribution across the world. out of the equation here, but like, how many people are willing to sacrifice their attention and the biased information coming to them just so they can—

Like, how many people are willing to sacrifice their personal context, right? So it was a guy in New Zealand reposted something on LinkedIn, and it was, um, it's from one of the UK police companies, uh, orgs, and it basically had a mock-up of, you know, 2 kids in school uniforms, and they tagged all the personally identifiable information. So, you know, the badge on the blazer says which school they went to, the mum holding or the dad holding, because you couldn't see their face, a polystyrene cup with their name on it from the coffee company, you know, John. Actually, I think it was a lady's name. Yeah, like a number plate for the car. Yes, all these things that were in an image that now, you know, bad actors can scrape. And I sit there going, you know, Number of people I know that put all their kids' stuff on Facebook constantly, you know, videos and images and basically document their entire kids' life.

Oh yeah.

And the kid's 5. In 20 years' time, right, when, when none of that information is private and it's available to any platform, what's that kid gonna say to their parents? Hey, thanks for that. Was it worth it? I don't know. Anyway, I go— I can go deep into conspiracy.

Yeah, that's my— I have a pretty hard stance on, on this stuff, and we try to like— anything that says like share our kids' pictures or whatever with like any big, you know, Apple or, uh, any of that, like we, we try to shut that all that down. I don't, I don't trust any, any of these guys. They'll make a quick buck, you know. They think they claim, you know, do good or whatever, but And I, you know, I am— I wouldn't say for— I'm not so crazy where I'm gonna wear a tinfoil hat and be like, ah, I can't. No, I would just like actively participating by like, you know, having my kids' pictures on like on Instagram or LinkedIn or anything. I'm just like, no, that's not where I want that kind of stuff to happen. And the other thing is, um, Yeah, I mean, we share our data so freely. I mean, I'm just like, right now I'm like on an absolute rager trying to get my phone numbers pulled from data brokers like crazy right now because I can't even use my cell phone. It's gotten so bad. Like, I just can't pick— it's just so many calls all the time and text messages. Like, if I sneeze on this podcast saying like virus software, I'm probably gonna get like, like just tsunami of communication. Um, so then it's just like, I don't know, it's so funny. I feel like it's gonna be like AI versus AI wars fighting like spam, right?

Oh, well, actually, Anna and I were talking about that when we caught up last week, uh, or the week before, and, uh, Joe Reese had jumped on. I saw him turn up, and, and Anna was talking because she's working in a new hot— yeah, new gig in a hospital, and it's greenfields, right? And so she's talking through um, the, the way she's dealing with that Green Cities program, right? And, and what she's looking at now, what she'll look at next. And I'm sitting there, I said to her, man, you're going to get snotted by salespeople, right? Because you've said that eventually you're going to buy or build something in the space. And so my joke was, that's fine, just tell them that they need to get hold of Joe, right? He's, he's in charge of buying for you. And then they'll get— so there you go, right? If you, if, uh, if you do need a virus thing, I think, uh, anybody that's Uh, scraping this, their AI should contact Joe Reese about selling some virus stuff. But let's go back to this idea of privacy, because— or context privacy— because one of the interesting things now, if we say that we can't reverse engineer context for an organization from the SQL queries that have been run, right, because we know that based on all the papers that have been written, human in the loop for that context is actually the most valuable thing you can do. And if you're sitting in an organization that has a history of shitcanning people when, uh, they want to save money or they want their share price to go up, and, and your comment about all your notes being on paper, right? Like, how willing would you be to put your knowledge into anything electronic and not just retain it in notebooks on your desk that you never share? Like, that's got to be coming, right, where people actually gate the knowledge they have because it's the only thing of value that the organization may not retrench their ass for. What do you reckon?

Yeah, no, I— it's so funny, like, you know, can you even trust your systems? Like, sometimes, like, you're reading, like, the terms of service and, like, when, like, people are like, well, I, you know, sometimes, you know, I do question Do some of these servers even need your full context? Can they get it from the shape of your information?

Yeah.

What I mean by that is the metadata, how it moves, how you're doing things on your system. Not necessarily so much that you're like, Tim Fraser opened Excel sheet 123 and looked at X column, but because of the sea of information, you could ascertain what how people are solving and doing things. So then like you're ultimately using these systems to train the system. You know, Meta is doing this for like tracking their own employees and having all the other kind of stuff. You know, how much longer are we going to be able to rely on the computers that we're using? Because the money is just— it's so worth, worth it, right? So yeah.

And you— I mean, you read that Spirit article, right, where Spirit Airlines— Google paid $10 million to get access to All the data in that organization, all the employee Slack messages, or every email. Now, in theory, the article said that personally identifiable data would be cleansed from that data set, or data lake probably, isn't it? Before it's handed to Google. But it's a liquidated company, right? Like, yeah, they— how much money is a liquidator going to pay cleaning that data before it hands off to Google? And If it's not cleaned, who gets sued, right? Like, if an employee's personal information is passed to Google as part of that data, um, and they find out, who do they complain to, right? Uh, but yeah, so again, if you now start to see companies that are in trouble having their, their conversations, right, being liquidated as an asset Oh, which is like, you know, if like Granola—

or I'm not saying to do, but like if that happens, holy shit.

Well, the, the company that came second in the bidding war, right, I think they bid 7-point-something million and lost out. They were a recruitment company. And so you're sitting there going, hmm, they wanted to pay 7 point $5 or whatever it was million US dollars for the data from an airline company, uh, with every conversation with every employee and every customer. Hmm, they're definitely not profiling people, right? They're definitely not building an engine, uh, to augment all their LinkedIn scraping data about Shane Gibson, uh, who just happened to be working for Spirit Airlines and, you know, Slacked all the time to his mates, his colleagues. God, my boss is a dick. Yeah, I mean, it's really interesting.

So like, you know, we're talking about just the sale of data and all that jazz. Could you imagine how many people when like 23andMe went bankrupt and they got bought by, uh, I believe private equity? And, um, you know, I was on there too. Like, just like the awesome power that you have with that is either like, honestly, I'm not smart enough to know all the information they could extract and glean for the details, but it's pretty powerful and personal information. And I was definitely one of the— I was like so fast to be like, thank God you're based in California and I have like CCPA. You're like, just dump all my data, dump. Like, I don't trust you guys whatsoever. I don't even know why I trusted them in the first place. I think that's just like, uh.

But if you look at the portfolio, right, the book that the PE companies hold right now of companies I've invested in that aren't profitable, that aren't gonna—

not—

they're not even gonna get a down round, right? They're just not going to get any more investment. Yeah, apart from the people that work for them who will leave, probably, right? Uh, the only other thing of value really is all that data about your employees and their customers.

Well, I think one of the things too is just like a lot of people are like— I think one of the things we're talking about, like, you know, the data and information that you have, like one of the things people are going to quickly realize in the data space is like, you don't want to sell— you don't sell necessarily sell your raw data. Don't like, don't do that. You're literally just literally giving the chicken away and keep keeping the eggs. The eggs only last a little bit. And, you know, there's gonna be a whole like network and we can talk, maybe we'll talk about this next time as I'm doing lots more fun research on this. But it's like, in the age of all this AI search and all the things that are happening, how do we get personalized experiences? How does it work with AI search? And you could argue we've had AI search for years. It's just Google's finally being like, oh, we have a challenger. And all that information, how's it going to be shared? Because I don't think it's going to be shared in the way that it was before, right? Where you have a cookie and a certain thing happens because you can't really share information like that. You're going to be sharing patterns essentially. And, and shapes of people. Well, Tim Fraser showed up and, you know, did this search. So people like Tim Fraser in the shape of what they're looking for, like these things, good or bad. I can definitely start to see that happening. Um, and, and it's gonna happen very, very, very, very quick.

Um, and, and how far does it go, right? So when I'm asking Perplexity a question, uh, and they've decided that they'll now influence the response based on who's paying them the most. Yeah, yeah, I'm gonna get, you know, uh, hidden editorials of, you know, I, I use Perplexity to figure out what cocktail I should make next. Yeah, it's going to start telling me to make rum-based ones or tequila-based ones, right?

Yeah, yeah, and I'll tell you, Grey Goose is the best vodka. And you're like, wait a minute, What is the— why did you determine that Grey Goose vodka was the best one? Tiny little like symbol, like asterisk at the top.

And why Grey Goose, right? I just said I need to make a cocktail, right? And it should just say vodka. Now why has it got a brand around it? So that's, you know, that's coming, right? I don't believe that it's not. But then if we're using those models corporately, Now what stops them influencing the models? You know, who's our top supplier? Oh, actually, you know, the data says it's 123, but it's a close call. I think we'll just influence it to be 256, right?

I mean, I, I would say you can't see the working, right?

You can't see— it's not deterministic, so you can't see the code that it's using to give you that answer.

Hey, man, it's a fair shadow, right? It's like, how do you— there's tons of research of trust and trust systems. It was already getting pretty bad if we want to talk about trust in the internet. We all know it. And the influencers, like what's real, what's not real. I mean, it's just going to be just monster. I mean, speaking of content and trust and black boxes, I think a lot of us have recently have been like, we all use LinkedIn, but none of us are like, yeah, whatever. It's just like, it's a, you know, it's like, I can't believe I'm going back to Twitter. Sorry, X, whatever that's called now. But like going back to like Twitter a little bit to get real information on AI development. I'm like, oh my God, I can't, I'm back again. Or, you know, I'm trying to use like Bluesky, but the network shrinking. I don't know. It's like, I I'm like, can I just have like a certified human network?

And does it have to be human? Like, because LinkedIn, you know, still is the only network that has reach.

Like, roughly, kind of roughly trust that people are verified. Yeah.

Um, to a certain extent, but it's gonna— somebody has to come out with a new network model, right? But it won't be like any of the others, and Maybe it will be just LLM friends and not real humans. Maybe that becomes the network, right? I don't know. I mean, I think it's going to get blown open. Somebody's going to come up with something creative that solves the problem because I think everybody I know that's on LinkedIn is like, it sucks ass. But where else can I go?

I mean, just like I did a post, you know, I mean, I post on LinkedIn quite a bit because I still think it's, you know, I'm a remote first person like you. You might live a little bit further out of place than I do. I live in Philadelphia. It's not notorious for its software engineering and data. dbt Labs might be from here and Vanguard up the hill and a lot of private equity and capital is here, but it's not known as a tech scene really whatsoever. It's not even in the top 100, I think, in the world. And so like one of my ways of connecting is through like LinkedIn and Substack and all those tools. And, but I was noticing like, it's crazy. So I'll do a post and I'll be on my former, you know, about myself, blah, blah, blah, a podcast episode, 500 impressions and 3 comments and 14 likes. I do one and it's like, 38 comments, 15 reposts, 115,000 impressions. And I'm like, is it because I did a meme with it?

Is that— I was gonna say, it's— I was gonna ask, it was a meme, right? All you did was post a meme.

No, it wasn't. It was like, it was an actual like moderately serious question. Maybe I hit like the smart things. I said, um, hiring and data analyst. And like, it's just like the keyword hiring is just like up to the right. It's just like we're AI, if you put AI in anything, it would just go crazy. And then, but because of these algorithms and your network effect, I don't know how it's working. And I try to understand it deeply. And I talked to other friends that have 40,000 or 50,000 people on LinkedIn and they're getting 1,000 impressions or 15 people are liking it. It's always the same 15 people in the group and they're like, I don't get this. I worked so hard on these black box networks, and now my content with my tribe like literally doesn't show up for them.

Yeah. And, and like, how often like are anybody AI slot botting, right? Like if you want to go against a competitor of yours, you just basically have an account that AI slots tags every competitor post, and that's going to hurt their ranking, right? Because that's the new punitive, uh, human in the loop thing. So yet you look at Substack, right? We've got Substack Notes, we've got what I feel is a more trusted network in Substack, but it's still not my go-to, right? If I want to see what my network—

Still private equity backed.

Yeah, true.

Is it going to be, you know, is it going to be the next Medium? I, I don't know. I, I like Substack. Um, I was on Ghost for a while, being more open source, but it doesn't have the network effect. It's just, yet again, it's easier to use, it's quick, it's got a good network. I don't know if it's gonna, you know, how much it's gonna change. Are they gonna say something that all of us are like, oh yeah, no, you're not really— you're still very much an American company, you're very much biased towards like getting pressured because somebody didn't like your content or something happened, and then you're like, oh, all right, that was quick. Yeah, I mean, thank you, Shane. I've had this— yeah, even though it feels like it was yesterday, but it's really been a month.

It's been a month, you know. dbt Labs have been sold, Orchestra's got funding, Databricks came out with, uh, a whole lot of new, uh, solutions and things. I mean, I, I listened to, uh, This Week in AI every now and again, right? It's like an hour to 2 hours of everything going on and Those, those, those 2 guys that run that band, they must do some research. Uh, so I, I don't think either of us have got time to do the research of what's happening every month. But yeah, I don't know, I think maybe, maybe we'll lock a little bit of a session in, right? Uh, part of what we talk about every month is who's done what, right? Because I lose track, and, uh, it's easy enough for me just to make a note on a piece of paper by my desk when I read something like, oh yeah, that's That's what's happened in the vendor land. I'll talk to Tim about that, see what he thinks. And then, uh, yeah, we can just do a little short synopsis of who sold who, who bought who, who opened something. Oh man, it's going to be like so much more trading.

It's going to be so crazy. Um, what conference in the next month or two, what conferences are you going to?

Uh, so just did the Data Community virtual one here. Not doing Big Data London this year. Uh, so the next one for me is presenting at Helsinki. Data Week at the end of October, which I'm mega excited to do because I've never been to Helsinki. And I think from what I've heard, Helsinki Data Week kind of is the closest to Data Day Texas.

Oh, very cool.

Which is, which is still my, my favorite one. I kind of gave myself a blanket. I'm not going back to the US until the next 2 years until there's some changes. And then Bloody Lynn Bender, who ran Day-to-Day Texas, has come out with a new unconference community thing, and I'm like, damn, I want to go to that. I think Joe Reese is running a conference in Salt Lake in early Jan. I'm like, damn, I want to go to that. But hey, I'm not going to break my principle. Uh, I won't be back in your country— well, sorry, that country— for a couple of years, so I'll just— I'll do Europe. Um, there's, uh, actually another one in Amsterdam in October I was quite keen to go to, but Can't make it timing-wise, but the other thing is it was €1,900. Oh, maybe I'm getting cheap, cheaper now. Maybe I'm too close to Scotland. Sorry, Scottish friends. But you know, two grand in euros for a conference now—that that feels like the old days, right? That just feels like too much. So yeah, what about you? You going to anything cool? Yeah, I'm going to.

Yeah, Lynn. Lynn's doing a smaller one in in Austin. Austin, so I'm gonna go there. I might be doing Big Data London. It's kind of like some timing things. Like, I usually would have bought the ticket a long time ago, at least the plane tickets, to catch up with people. Just need to see kids, kid and family stuff going on, so need to see. And then I might be going to San Francisco for the— there's a Mother Duck conference in November. And I think that's about all the travel I'm gonna be allowed allowed to do.

Uh, oh, actually, uh, Johnny Winter and I are talking about doing, uh, a low-key data meetup, uh, for Christmas on this side of the pond. So, uh, early December. So, you know, maybe, uh, maybe get— yeah, come over then. That's probably not perfect for you given you probably need to go see family at Christmas. But, um, yeah, we're just kind of toying with, uh, maybe, uh, a small data community thing in the UK would be a fun thing to try, try out. So, uh, yeah, we're gonna go finalize that tomorrow, whether we're going to do it or not. Um, as I said to Johnny, might just be him and me in a pub somewhere in the UK, but hey, I'm okay with that.

All right, Shane, thank you so much for your time.

Good to see you. We'll catch you up, uh, in a month. All right, catch ya.