ShagilityAgile Data Coach, Author, Agile Data Guides · Your Agile Data Guide

Store business logic outside tooling #substack #shorts

11 Jul 2026 · 1 min

Watch on YouTube

1:01

In this short monologue, the speaker contrasts the architectural roadmaps required for very small (3‑person) data teams versus larger (5‑15‑person) teams, arguing that each scale demands a distinct stack. He predicts that current data platform leaders like Databricks and Snowflake cannot remain unchanged; they will either grow through consolidation, be acquired, or evolve, making their present forms unsustainable. Consequently, he advises keeping all business logic separate from any specific ETL or data‑tooling platform, storing it in a canonical source and using smart scripts to hydrate whichever tool is in use. While native, instant migration between ETL tools remains unrealistic, the rise of large language models (LLMs) is dramatically speeding up the ability to move logic across tools, improving portability and future‑proofing data pipelines.

Chapters

you know, the stack or the roadmap or the blueprint for an organization with a 3-person data team versus one with 5, 15-person data teams is very, very different. So I think you've got to fit. And then my view now is, because we know that Databricks and Snowflake and those platforms, right? Like they, they won't survive in the form they are right now. Yeah. Nobody ever does, right? So they're either going to get bigger or they're going to get bought. Or they're gonna die, which is probably unlikely now. But what I would now start focusing on is holding the context of all my work outside of the tooling that I use. So I would find ways of storing the business logic completely separately and then using it to hydrate whatever tool I'm using. So I store it over here and then I write really smart stuff that then hydrates it into that tool. Now, you're not gonna be able to natively go from one ETL tool to another in the drop of a hat. But the trends, the ability to move between them now, especially with the LLMs helping us, is, uh, is a lot faster.