Build Data Products Without A Data Team Using AgileData

Hello and welcome to the Data Engineering Podcast, the show about modern data management. Atlin is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlin's active metadata capabilities. Push information about data freshness and quality to your business intelligence automatically scale up and down your warehouse based on usage patterns and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com atlin today. That's a T L a N to learn more about how Atlan's active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork and Unilever achieve extraordinary things with metadata. When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends at Linode with their new managed database service. You can launch a production ready MySQL, Postgres or MongoDB cluster in minutes with automated backups, 40 gigabit connections from your application hosts and high throughput SSDs. Go to dataengineeringpodcast.com linode today and get a $100 credit to launch a database. Create create a Kubernetes cluster or take advantage of all of their other services and don't forget to thank them for their continued support of this show.

Your host is Tobias Macy and today I'm interviewing Shane Gibson about Agile Data.

A platform that lets you build data products without all of the overhead of.

Managing a data team. So Shane, can you start by introducing yourself?

Yeah, hi, I'm Shane Gibson. I'm the Chief Product Officer and co founder of agiledata IO and my longtime listener and second time caller.

Yeah, and for folks who didn't listen to the previous episode you were on, I'll add a link to that where we discussed some of the ways that Agile methodologies kind of be applied to data engineering and data management. And so today we're focused more on the product aspect of what you're building. Before we get into that, for folks who didn't listen to that past episode, if you could just give a refresher on how you first got started working in data.

Yeah, so I started my career out working financial systems. So in the world before enterprise resource planning or erp, when we had separate accounts payables and, and accounts receivable type modules, so I was kind of in the systems accounting land looking after those financial systems and from There I jumped across into vendorland. So started working for some of the big US software companies, but based out of New Zealand. And as part of that realized that my passion really wasn't in those erps, but I really liked the idea of data and analytics and back then it was really cool. Business intelligence, it's how old I am. So did that for probably about a decade and then from there jumped into founding my own consulting company. So typical data and analytics consulting company, 10 to 20 people on the team and we go out and help customers implement platforms and strategies and do that cool data work. And from there I morphed into agile coaching. So I found that I had a passion for working with data and analytics teams that were starting their Agile journey and coaching them on ways of working, so applying or helping them apply patterns that I'd seen from other organizations or teams that have worked with particular context and helping them apply it in their way of working and see that success of that team really just starting to rock it, enjoy their work, deliver stuff, get value, get feedback. So as part of that, about three and a half years ago, my co founder Nigel and I started Agiledata IO. Taking a product focus to that kind of capability.

Can you give a bit of a high level on what it is that you're building and some of the ways that that product is manifesting and some of the target audience that is driving the focus of how you're building it?

So we're combining software as a service product is the way we think about it, that manages the whole data process, from collection through to ready for consumption with those agile ways of working. So we believe that any product should be bound by ways of working. You can teach people that make sense, right? So not a methodology as such, but a set of good practices that make sense when you're doing a certain part of that data supply chain. So that's our focus of building out both this way of working that's repeatable and teachable with the product that supports it and makes it easy. In terms of the audience, we kind of break it down now to two audiences. The first audience is the buyer who would actually buy our product capability. And that really is anybody that's got a data problem. There's a bunch of people that are in organizations that know they have data, they can't get access to it, they just want that data to be turned into information so they can make a business decision and take action, and then from that action get some outcomes and they struggle to get that done. And there's different Reasons for different organizations, why that happens. So that's the buyer, the user. We are hyper focused on a data savvy analyst. It might be a data analyst, might be a business analyst. It's somebody who's data savvy but actually gets frustrated with the fact that they have to hand off the data work to an engineering team and wait. And it's not the engineering team's fault, right? They just got too much work to do. But the analyst actually wants to get that work done. And so if we can enable them to do that work themselves, give them that self se in that data management space with the right guardrails, so it's not ad hoc, then we think that we can help those people get that information to their stakeholders quicker, better and ideally with a bit more fun.

In terms of the types of organizations that are likely to use the agile data platform, I'm curious if you can give some characteristics of maybe the size or scale or some of the ways that their technical capabilities or engineering teams might be structured or the skill sets that they might have, or maybe even more notably the skill sets that they might lack on the team.

So currently what we find is customers that don't have a data team are the ones that we can help the most. So they are typically somewhere between 2 to 50 or 100 people. They have started either building out their own platform software as a service, kind of internal capability. So they're a startup or they're an organization that is using a bunch of different software as a service products to run their business. And they're starting that journey of okay, we've got all this data and now need to put it together when we look out in the market. To do that, we now have to hire a bunch of data people, we have to hire a bunch of data engineers, we have to buy a bunch of software or software as a service. We have to cobble together this modern data stack and that's expensive and it takes time. So they are currently our customers that we serve. Our focus in the future is really enabling organizations that have those analysts in place and then have a constraint around the engineering practice and therefore that whole idea of bringing self service into that organization. I kind of liken it to the wave we had previously around self service, BI and visualization, the tableau kind of wave I call it, where there was a constraint of the ability to create reports. We used to use reporting services or business objects. It was a technical product, there were a bunch of gates to use it which were there in place because you had to have competency around a whole lot of things to make that stuff work. And we then saw this wave of self service capability come in in that visualization space and we enabled more people in the organization to do good work. We see that same wave starting to happen for data management and we believe that the same self service capability can be brought to that audience and enable them to do that work, freeing up the engineers to do the work that they are really, really good at and should be focused at.

For teams that maybe have an existing data engineering group or a set of data professionals, what are some of the ways that they might interact with the agile data platform and some of the maybe support burdens that are alleviated because other stakeholders and analysts in the business can just use agile data to be able to perform the workflows that might otherwise require the intervention or support of those data engineers or data professionals.

Typically, what happens is data engineers like to code, they like to build things out themselves. That's what they're trying to do, that's what they love doing. So we often find if there's a big data engineering team in an organization, we probably aren't good for them because culturally they like to build rather than buy or lease. If we take that away and we look at it, we look at it as there's a bunch of plumbing that everybody has to do for data. If you think of it as plumbing, you think of it as moving water. There's a bunch of pipes that you always have to build. And our data engineering teams are all organizations are stuck with building that plumbing and spending that time day in and day out. So what we look at is if we automate that plumbing for them, right? If we take just that pure movement of water from the collection through to ready for consumption, then the engineers can focus on the more fun stuff. Now one of the areas that they still need to focus on is around data collection. Because when you talk about a big organization, especially one that has a bunch of data engineers, they typically have a myriad of source systems or systems that capture where data goes into. And some of them are easy to get to. Software as a service products like Shopify or Salesforce, where it's fairly open to get the data, but nine times out of 10, there's also a bunch of proprietary firewall, whatever on prem or private cloud capabilities that we can't connect to. So there's still that data collection work, right? To be able to pick up the data and make it available, to be ready to be transformed and consumed. And then if we think about we take care of that plumbing bit, then once that water turns up at the end of that pipe, that's where the real value a data engineer can add, or an engineer in general, because engineers are good at solving problems. And so if we think about that water comes out of the tap, it's nice and clean as much as it can ever be. Now we have to make that data useful. It's consumable, but it's not useful. It's still plain water. So how do the engineers then get involved in the business problems that the organization has around how to use that data? Is that helping the data scientists make better machine learning models which give better recommendations to their customers? Is it focusing on some things like master data management and the complexity of what are the actual rules we need to combine that data because the keys aren't the same, there's still 101 problems to be solved, but the problems aren't moving data from left to right. That's the bit we take care of. So we still see the engineering skill set as having massive value in every organization, but we also see the engineering skill set as being one of the primary constraints for organizations right now to get data work done. And that's the bit that we want to automate for them.

So as far as the design aspects of how you think about building the agile data platform, given the focus on these analytical workflows and on organizations that don't have a large footprint of data engineers, I'm curious how you think about the design and the interfaces that you want to build out to make these potentially very complex workflows understandable and manageable by people who don't necessarily want to make that their entire job. They just want to be able to get something done.

So we think about it in two ways. So Nigel, my co founder, is the techie, he's the plumber. He's the guy with the experience building these things out over generations. So we think about complete automation in the back end every time. Right. We think about if people are using this product and we're not there to watch it happen, how do we make sure it's bulletproof? And as part of that, we become highly opinionated. So for example, we have baked in a pattern in our product where everything is historized. Every time a new record comes in, we check to see whether it's a change, and if it is, we store that change. We don't have a conversation about do you want any CD1 or 2 or 5 or 6 type behavior? We go, actually, we're going to automate it. So every change comes in, it's stored as a change, and that history is immutable. And by being opinionated with that, we take away some of that complexity from the user. Now, there's a hell of a lot of complexity on our side, right? Because if we're getting fed snapshot data, we've got to do the delta differences. If we're getting event streaming, we've got to figure out that it was an event change or a new event record. There's a lot of complexity for us that we need to plumb under the covers. But that's our job. So that's the first thing we do is wherever we see a pattern that had complexity when we were consultants and delivering that for a customer, then how do we automate that in the back end and make it as bulletproof as we can. The second lens we then take, which is the lens I take is around the product, right? So how do we get this interface, this app, and how do we make it available to an analyst where they're building out data, stuff that is typically complex and making it easy for them? So an example around there is data design. So we are really strong proponents and highly opinionated that data should always be designed, it should be lightly designed. So how do we take this data design process, this data modeling process, and help an analyst do it in five minutes rather than take six months? And how do we give them tools that allow them to do it where they're not having to do ERD diagrams, entity relationship diagrams, and we have to teach them about ducks feet and many to many joins and all those kind of things. And so what we did was we prototyped and tested this idea of design canvas where they can go in and say, I have some cool concepts, customer, product order, I can drag those tiles to say, there's an event, customer orders product, and that actually builds a conceptual design in our product? And then, okay, now I've got a conceptual design. How do I populate it with data? Well, we've built a rules engine that uses natural language, right? So given this data is in this history tile, say shopify customer. And we've got these fields, you know, name, date of birth, those kind of things, then populate this detail tile about that customer. And then when new data comes in, automatically trigger a load, right? And trigger the change detection. And, you know, somebody's changed their last name, bring in a new record that there was a change on this date, that then their name changed. So, effectively, by working with analysts, we look at some things that we used to do unconsciously as engineers and data specialists, and we try and figure out how we build interfaces that make it really simple for them to do that work without knowing the complexity that lives under the covers. And to be fair, that's hard. Every time we do one, you have to sit back and go, okay, as humans, what do we do? What are we looking for? What are the things we look for that are triggers to make a decision that we need to do a piece of data work and therefore can we automate that so they never have to do it, or do we need to prompt them with some decisions and by making a choice of yes or no, we do that work for them? Yeah. So that's our focus. Right. How do we bring that interface to an analyst where they do the work without knowing they're doing it?

To your point of not forcing the analyst to understand or have to go through the process of educating themselves on these various different types of joins and how to find the appropriate data sources to be able to answer their question. I'm wondering what are some of the, I guess, thorny engineering problems that you've had to work around to be able to make the end user experience simple?

Every one of them is thorny. Every time we go and do something that simple and we're hyper focused on removing the effort or removing the clicks. So I'll give you an example. We had a customer that we were doing a data migration for that was a use case and we built the ability to bring in some data from something like Shopify, land it into what we call our history layer. So that's the immutable layer that holds all data over all time, all changes. Right. We really care about that layer and then creating the rules that we use to populate a concept. So that's a customer, that's a product. We built all that out and I got really good at it. I could go and do a rule for that. Normally about two to three minutes it was in production and the time was looking at the history data, understanding where the key was, creating the rule, running the rule, going back, having a quick look at the data, making sure the test had passed and that the data looked right. So I got it down to three to five minutes and I was like, yeah, that's great. At the time, we had to write a rule from the landed data to the history data. So I had to manually go and do those rules and you bring in Shopify data. It's less than 25 tables, three minutes. I'm like I'm a machine. For this data migration system though, we had an on premise SQL Server database we had to bring in and because we were migrating the data, we had to bring it all in. So there were just over 700 tables now therefore I had to go and create 700 history rules, two to three minutes, a rule that was a couple of days of my life that I did not enjoy. So we went back and we said, okay, how do we automate that? How do we drop in 700 tables and get those history tables tiles created automatically without us? And so we went and automated that. And so now that's what happens, right? So every time we do something and it takes us time or we have to make some really complex decisions, we just go back and think about it. How do we refactor that, how do we make that simply magical. But that is hard. Every time we do something as an engineer in data, there is a hard problem to be solved and we can't underestimate that.

In terms of the actual implementation of Agile Data. I'm curious if you can dig into some of the architectural elements and some of the technical aspects of how you've designed the platform to be able to slot into an organization's existing stack without having to re engineer everything around Agile Data.

So the first thing we did was we had to pick a cloud platform. We had to make a decision. We're going to be multi cloud or single cloud. And we decided that as a software, as a service product, which was our goal, our vision, we wanted to pick a single cloud platform. And in theory the customer shouldn't care. Actually they do. It's amazing how often the customer cares. But in theory we're just a software as a service platform. As consultants, we'd worked with Microsoft and Azure for a long time. We'd done what I would call now the legacy data warehouse cloud platforms, the redshifts, the Microsoft pdw. And we knew we didn't want to use those. We knew that they were, they were great at the time, but they had been cloud washed and there was a whole lot of problems that we'd encountered. Vacuuming out your redshift tables was always a nightmare. And at the time, this is three and a half years ago, Snowflake really was the up and comer. So we had planned to build our entire platform, our entire product on Snowflake. At the time I was coaching a data analytics team for an organization and they were starting to build out their platform and they did a shootout between BigQuery and one of the other vendors and I'd never seen BigQuery before and so I was lucky enough to sit through that whole process of evaluation and at the end of it I said to Nigel, look, I know we're going to go with Snowflake, but we probably need to give this bigQuery thing a bit of a bash because it looks like Snowflake, sounds like Snowflake, but something weird about it, it just seems a bit different. So we did a proof of capability around it and we found that actually there were some things in there that we really liked. And so we started building off on lit query and that was we got a whole lot of unintended consequences out of that. Saved us years of engineering effort. And what it was was that by going into BigQuery we went into the Google Cloud infrastructure, into their ecosystem and there are a bunch of services that are available in there that we keep using. We keep adding new things we need to our platform in the back end that we leverage Google Cloud services for. And the majority of their services are serverless, which means the way we pay for it is amazing and it's pre integrated. So when we pick up a new service it just tends to work with all the other Google services. And the last thing is they're highly engineered. The amount of engineering thing effort they put into their products is amazing. I mean their marketing sucks, right? As a partner of them.

In my experience running a consulting company, the other cloud vendors and the other product companies are so much better to work with. But in terms of Google Cloud the engineering is amazing. The downside is you have to be an engineer to use it. You have to be a Nigel, not a Shane to configure a platform on Google. But that's what we do is we build the easy bit on top. And so from my point of view that decision around using Google Cloud was an undid intended consequence when we got a massive amount of benefit out of it. If we then think about our platform, we made some really big choices upfront. So serverless only. We would only ever use serverless capabilities unless we had no choice. So no containers, no kubernetes and that's had a massive amount of benefit for us. API everything. So we have this idea of a config database that holds all the core logic for the data that's fronted by a set of APIs and our actually sits on top of the API. Those APIs are secure but open. So we're really intrigued about whether customers will actually start using the API and not the App at some stage. And that's had some massive benefits for us as well. By decoupling the data from the config, the config from the API and the API from the app. Right, that kind of standard software engineering for everybody else. But Nigel and I aren't software engineers. It's not our background. We came from the data space. Then the next thing for us was volume. That trade off between over engineering everything to take a volume of users or a volume of data versus getting something to work and then dealing with the volume problem when it hit us. And it comes around that idea of knowing where your data is, knowing that you're going to have a problem. So for example, when we were bringing in Shopify data for one of our first customers, the numbers were tiny. When we started doing event data from a customer in a clean room use case, you know, we were getting somewhere between 40 million and 3.5 billion road changes or road transformations a day. You know, we had to refactor some of our stuff to A, keep the cost down and B, just make that thing bulletproof. But we knew we had to do it. We just waited until it was a problem and then we went into hyper engineering mode to solve it. Right. We'd already designed the patterns in our heads, right. We'd done what we call agile texture. So we'd done some pictures in some mirror boards about how we would solve it it as a guess. And then once we had that problem, we ripped into saying okay, will that solve it? Do we need to iterate it? So that idea of incrementally building it out just in time, but still doing agile texture. Right. Still figuring out we've got this problem coming up in six months, if we're successful, what would we do? And not backing ourselves into the quarter where refactoring it was horrendous.

As far as the overall integration path for using Agile data, what is some of the technology stack that an organization needs to have in place for them to be able to take best advantage of what Agile data provides and what are the options for being able to extend the set of integrations that Agile data will be able to work with?

Yeah, so if we think about it as left and right, left ring data collection and right being data consumption. So the first thing is we collect the data and we store it it so we're not a virtualized warehouse. So that's the first thing for our customers. If they're uncomfortable us holding the data securely for them, then we're not the right solution. And then we looked on the left around data collection and I thought that was a solved problem when I started this. It's a solved problem, right? We don't need to deal with it. And boy was I wrong. So we have built out collection patterns, we call them, as we get new customers that have new problems. So there are a set of patterns around software as a service, app, apps. If you've got Shopify, Zero, QuickBooks, Salesforce, something that has an API, there's a bunch of patterns for going and grabbing that API data and bringing it back and collecting it into our history layer. The next one we had was we had a customer that wanted to file drop, right? They had a bunch of ad hoc type data, but it was repeatable, if that makes sense. So there was no system behind it, but we needed to get that data in. So we built a file drop capability, right, where they can go and upload the data, CSV or Excel or JSON and then that manually just file upload and then we take care of it from there. The next one was we had a customer that wouldn't let us go in and actually touch their system. So they wanted to push the data to us and it was event data, right? So they said, okay, we want effectively demilitarized zone and we're going to connect to you and we're going to push the data to you when we feel like it. So we had to build that pattern out. How do we actually have automated file drops? How do we do it based on event data? How do we trigger the fact that they turned up? How do we deal with the fact that they're always going to give us data they've already given us? Because no matter how much it's automated, eventually you get that file again or you get a file with overlapping where half the data we've seen before and half we haven't. How do we engineer for that? Then we had one, course they have one where it was on premise. How do we actually go into an on premise database and pick that data up and bring it back? And then we had one which was a clean room. How do we actually have, in this case, 38 different companies, each one of them using a different collection mechanism to give us the data and then make sure all of that comes in on a consistent basis every day so we can mesh it up and provide that single view. So again, I thought data collection was a solved problem and I was wrong. So that's the left, right? And then the right hand side is data consumption. So we don't do that Last mile, we looked at it and we said if you look at that last mile space, visualization, dashboards, natural language queries, analytics, you know, machine learning, table stakes to play in that would be three to five years of our lives. And we're bootstrapping, not venture funded. Right. So we're very focused on where we spend our time. So for us, one of the benefits BigQuery gave us was 9 out of 10 data consumption tools. Those last mile tools talk to BigQuery. So as long as you can talk to BigQuery, you can use the data that we make consumable for you. We have an API layer. In theory, the Last Mile tool can consume the data via API. But what intrigues me at the moment is I struggle to find a last mile tool that queries Data based on APIs that isn't a data warehouse pattern, that isn't taking that data and then storing it again to make it consumable. And so I'm really intrigued to see whether that's going to change, whether the market's going to move to have last mile tools that are truly API focused.

Yeah, it's definitely an interesting space and there actually has been some growth there with things like the embedded analytics use case. Some of the companies that are operating there that come to mind are things like Cube JS or I think they've been renamed to just Cube so that they're not tied to the kind of JavaScript aspect. I think it's maybe good. Data and Sisense I know are also focused in that embedded analytics case. And then there are projects like tinybird that build on top of clickhouse. You can use their data as an API so you can embed it into some products. So definitely an interesting aspect of it as well. And to your point of building on top of BigQuery, I'm curious if you've had to construct any sort of custom caching layers for being able to maybe pre compute or pre aggregate requests that users are making frequently so that you don't have to Go back to BigQuery every time and pay either the latency costs or pay the extra query costs all over again because somebody happened to query it two times, five minutes apart.

So that's a really interesting space right now. So we're waiting for the metric or semantic service, the Looker ML core out of Google because we need that at some stage soon. But till now it hasn't been a problem. And so what we do, right, so effectively the data comes into our product, into the history layer. Right? History layer looks like the source system, but it's time series, right? All changes over all time and that's immutable, right? We never lose that data. And then we have an event layer which is modeled, right? It has the idea of concepts, details and events and that's in the middle. So it's a typical three tier architecture. And then our last layer is our consume layer and we denormalize that. We actually have big wide tables. Now we can do star schemas, right? We can actually dimensionalize the data. We just never had to. Now the reason we do those big wide tables is when you actually talk to an analyst, they just want to query a table. Yes, they can do joins, but they don't want to. Why should they have to? Effectively we give them big wide tables. And the good thing about BigQuery is it just eats it in terms of volume. We've got some consumable tables there that are hitting 100 million, 200 million rows now. And BigQuery just eats it. The latency of response is seconds, it's not sub second. There's a brilliant analyst out there in Twitter world called MIM who's doing lots of really cool research on his own time around latency and things like DuckDB and those cool things. And if anybody wants to see somebody doing some cool stuff and sharing, go follow them. But we're not dealing with that sub second response use case. So most people are willing to wait a second or two for that data to come back. And most last mile tools actually have a whole problem of getting the data back and visualizing it. There's a delay there. Anyway, what we found is we actually introduced the BI engine into the architecture for us purely as a cost saving. So with BigQuery there is this idea of cache hits. So if you query a table and that query gets the results, that query gets persisted for 24 hours and you don't. But if you watch what a user does when they use a dashboard or report, they're filtering, right? They're constantly hitting filters and therefore those case shits don't happen. So we've introduced BI engine in the middle and that now keeps our costs down because that's doing effectively in a memory olap queue. It's rudimentary at the moment in terms of the way it deals with that, but it's good enough for us right now. Now as soon as they bring in the looker ML core, that semantic layer, that metric layer, we're all over that. We've done all the Agile architecture for it. We've done the UI design work for it. We just need that service right now because we want that extra layer of caching. Right. To give us some benefits and bring the metric out of the last Mile tool back into us. Right. So that's a core feature that we need to save some of the complexity. But yeah, right now BigQuery just eats it. I mean it really is kind of amazing how lazy you can be with some of the things that we used to worry about 10 years ago where our databases couldn't handle it or they were too expensive. That's one of my learnings for the last couple of years.

Yeah, I brought up the question about kind of the caching and pre aggregates because a number of years ago, I think it was maybe six or seven years ago at this point I was involved in a project where I was trying to generate user facing analytics off of data that we were loading into BigQuery and the multiple second response times were a non starter for that purpose. So I actually had to pre aggregate the data out of BigQuery into Postgres so that that could act as sort of like the OLAP layer. At the time I wasn't yet experienced enough to really be able to think quite in those terms and design it effectively, but I got it to work.

Yeah. And because we've been around in the market for a long time, we remember the OLAP MOLAB ROLAP days and then the horrible disconnect we had between our relational data warehouse and our OLAP cube and making sure they were synced and refreshed. The other thing is, as I said, we always do an agile texture about things we worry about. So we knew we had to worry about performance, we knew we had to worry about cost. We have a whole lot of what we call levers, we have a whole lot of patterns that we haven't implemented yet that we know we will need to at some stage. Things like materialized views, at some stage we know we're going to need to bring those in and use them early in the Google kind of product life cycle. They're fairly basic compared to materialized views that we used to have in the Oracle days or you know, the teradata equivalent. So we particularly won't invest in that layer until we really, really need it because we get the benefit from Google of all the engineering without us having to build it. The number of times we found a feature from Google turn up just in time that we didn't have to build it ourselves has been amazing. But there's been a Couple examples where there's some early stuff came out of Google that we really wanted that didn't make it into their production and we're like, we really want that feature, right? Would be save us money or save us time or be awesome for our audience, our users. And then it just didn't make it right. And so we've got to be really careful what we invest in. Another area in terms of that caching that we're keeping a really close eye on and we're going to have to make a beautiful decision on at some stage is DuckDB. Right? This idea of taking a subset of that data out as a wasm into the browser and using that to provide immediate response to our users in our app. We think that is an area we'll probably invest in, right? Because that feedback of being able to see, here's a piece of data, here's my rule for transforming it, here's the impact of that change without clicking a button and waiting for that query to run. We think that immediate feedback to an analyst in terms of designing the rules that transform data, we think that'll be magical as well, right? But we just got to decide when we're going to build that out or whether Google's going to give us a service that does it for us.

Prefect is the dataflow automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn't get in your way, Prefect is the only tool of its kind to offer the flexibility to write workflows as code. Prefect specializes in gluing together the disparate pieces of a pipeline and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, prefect power is over 100 million business critical tasks a month. For more information on Prefect, go to dataengineeringpodcast.com prefect today that's P R E F E C T.

Another interesting aspect of what you were discussing is the question of data modeling for that end user analyst and also some of the ways that you think about the semantics and documentation of the attributes and the columns that are available in the modeled tables and wondering if you can talk to how you think about that when you're building this as a service and you just want to be able to say to the analyst, pull in the data that you want, write the queries that you want, we'll worry about the rest.

So the first thing is we Think about documentation as something that should happen, but it should happen at the right time and then it should be actionable. So one of the first set of features we built out was a data catalog, because we believe data catalogs should be embedded in every product. Right. This idea of being able to see what the data looks like, what the profile is, have context in terms of notes against it, see the lineage and the rules. Right. Every data product in the world should have that bundled in it. Should be table stakes. You still need an enterprise one if you've got multiple platforms and multiple products, right. And you want to combine those together. But every product should just do this as table stakes. Looking back, I probably over engineered that. It was a personal one of mine, had massive value, but I probably added a whole lot of features that I don't think will ever get used. And so again with my product hat on, yeah, it's going to be really interesting to see whether I'm brave enough to remove them. And time will tell, right? But what we know is documentation should be done at the time we're doing the work because we never go back and redo it. So if I'm creating a rule and I need some context about what that rule's doing, then we should create documentation. And what we did was as we built out our rules engine and the interface for it, we did a whole lot of prototypes. I had an idea in my head of what we're going to build and it wasn't what we built. What we actually ended up building was almost notebooks, almost Python or Jupyter notebooks, as in there's a line at the above that has the data coming in. We call that given. So given this data and then there's a bunch of and statements, right? And I do this and so this data's coming in. So let's say I've got a table that's got party entity in it, right? I've got a history table that's got employees, suppliers and customers in the same table from the source system. Because the software engineers found that's the most effective way to build out their system. So in the rules engine screen you'll effectively go in and say, okay, given that table and the type equals customer filter and maybe the customer has had a transaction in the last 30 days and the customer's not deceased, then create a flag called ActiveCustomer as a piece of detail. And so that idea of actually just adding rows like a notebook we found, it's been particularly successful. And then what we said is, okay, well, doing that, we need to put some notes against it. And why do we? Well, because what happened was I do a piece of work for a customer, I create a rule and then I go do some work for another customer. And six months later the first customer asked me to make a change. Right? So I go back into that rule and I look at it and it's very simple to understand, right. Given this. And, and, and then so I know what I've done, but I don't know why. Why am I filtering out that record? There's gotta be a reason that I filtered that record out. Like I wouldn't do it if I didn't need to, because I'm lazy, Right? But what happens if I remove it? Yeah. And so what we found is putting notes at each line means I can do a really simple statement, filter this one out, because there was a problem with the source system six months ago. Therefore I've got to remove those records or change this one because for some reason the type coding was out of sync for a little while. And the easiest way to clean that up is to do it in line. And so by putting those little notes in, I get the documentation I need at the right time to take the action that I want. So what we found is those in place notes are really valuable. The data catalog thought had high value. Time will tell. But yeah, every time we find an action where documentation would have been useful, we then go and figure out how we add that documentation. So one that we've got to add back in, which is really interesting thing for me was we version our rules by default, so it's effectively git behavior. So if you go in and change a rule, you have no choice. The previous version of that rule's stored for you. There's no git pushy, pully kind of stuff. Right. It's just in the background. Every time you modify something, the previous version is stored because that's good practice. What I don't have is a box to say why I made that change. So we're going to add that piece of documentation in because I'll go back in three months and go, oh, I've got two versions of the rule. Why did I version that rule? And so again, it's not until you use it in anger for a while, you go, oh, that piece of documentation would be highly valuable. I'll type it in as a git, you know, effectively my git note. And so we have to go back and add that. And so for me, that's what documentation's about how do we edit it at the right time, that it's actionable and it's actually going to be used, otherwise we're not going to fill those boxes out and they'll be blank.

Yeah. And you talked a little bit about the kind of discovery aspect with the data catalog question. And I'm wondering what are some of the, I guess, benefits that you've seen of biasing towards the wide table in terms of the data discovery question and some of the pieces of feedback or challenges that your customers have run into as far as being able to understand what pieces of information are available to be able to factor into their analyses and then digging more into that semantic aspect of understanding. This column looks like it does what I want it to do. How do I understand that? It really does.

Yeah. So let's talk about the cool stuff and then let's talk about the hard stuff. So one of the cool things that again was a surprise to us how often we use it now is Google introduced in BigQuery, the ability to search for a set of data. So you can go into a table, you can write a query and say, here's a word. So let's say the example I use in our demo, right, I've got a table of every car in New Zealand from our transport agency. So every time a car's registered, they publish the details of that car. And so, you know, there's 4 or 5 million rows in there for every car I can go in. And I'm a big fan of Minis. And in the Mini world, there's a race version called John Cooper Works or JCW. So when I've been BigQuery, I can go in and say, search this table, this history table of cars for the word jcw. And it will search every column and every row and bring back a subset where it has that center letter. So we put that into our app, right, because it was a freebie. Okay, well, it's cool demo, right? Go in, search. What we didn't realize is how often we use that. So, you know, an example would be when we first work with a customer and we're building out a business rule and that rule's reliant on a piece of data, right? It's a filter or query that says actually, you know, subset by this, or calculate it based on this field. And they use a term which is the word in the app, but that's not the word that's in the data. Right under the covers. The engineers have given it a completely different name for that field. And so we've got to find that piece of data to be able to search. And so yeah, what we used to do was say, give us a screenshot and then we go and look for it. And then we'd talk to the engineers and say, okay, you see that field there? Well, what's it in the database? What we can do now is say, give us an example of that data. And then we go into the history table, we search for it, it and it comes back with those rows and we go, okay, we think it's that column that saved us so much time. Another example is validation. You know, got an example where a customer goes, look, I'm looking for this order number, blah blah, blah. And I can't find it. So we just go into the catalog and we go into, you know, either start at the left or start at the right. If you go into the consume table and search for it. Ah, it's not there. Damn. Okay, let's go into history, search for it. Yeah, it's there. Okay. It's a rule problem. Yeah, the data's turned up, but my rules are excluding it. And then you go into the rule and find the rule and see why that one's been excluded. So that ability to search actual data has, we've used it time and time again in a ways that we, we didn't think about it. So there's some cool stuff, right, that has value. One of the problems we still have is the last mile tools. It's still incredibly difficult to be in a last mile tool, a dashboard, a report, an analytical tool, and see the context of the data. Right. And we haven't solved that problem at the moment. You have to go back into the catalog, then you have to search for it, you know, which consume table you're using.

Right.

But then you have to go search for it and say, okay, what's that field? And then look up the config and say, okay, what's the logic? Or the thing? And so that bundling of that process back and forward would be great. Now we're seeing a lot of the last mile tools now start to do that with add ins to the browser. It's kind of interesting. It's almost like the browser's taking care of the linkage between your metadata app, your catalog and your bi tool, your, your last mile tool. So we're keeping an eye on that space to see if that's where we should play. Should we open ourselves up to that kind of pattern. If you talk to an analyst, that's what they Want I'm in this report, I can see a number. Just tell me how you gave it to me. Don't make me go away somewhere else and find it with 600 other clicks. But I don't think that's a solved problem yet.

For people who are interested in being able to adopt the agile data platform, I'm curious what the onboarding experience looks like and some of the ways that you have iterated on that process to make it easier for people to be able to adopt and adapt to agile data as a way of being able to build their analytical use cases.

So we're going on a journey, right? We gave ourselves seven years to be an overnight success. And so our first three and a half years have been focused around that buyer solving a business person's problem to get access to their data and being a technologist. Right. And especially having worked for big software companies before where I was in what was called pre sales, which was system engineering or customer experience, I think now where my whole job was to work with the sales salesperson to present the product to the customer. So they said, yes, that looks like it's going to solve our problem. Can we buy it please? I naturally want to go into product demos. And so that was one of the mistakes I made really early on was I was talking to somebody who had a business problem and I wanted to show them the cool features we built and they'd turn around to me and go, why should I care? That's not what I'm buying, right? What I'm buying is you're solving my data problem. I'll give you some money. Me, you'll get my data, you'll give me some information, I can make some business decisions. I don't care how you do it, just make it go away. So that whole messaging around that has been really key to us is that actually we've got to bundle in both our platform and our services as a fixed monthly fee to just make that problem go away. The second half of our seven years though is focused on the analyst, right, about that software as a service. And that's what we're working on the moment is how do we actually onboard them them and get them to be able to come on and use the software as a service platform or product to do the work they need to do without having to be data experts. Now they have to be data savvy. They have to understand how to troubleshoot and what data means and what a concept for data is, what a customer versus a product is. They have to understand that because all data is complex, but how do we onboard them where they can just do it without us? And that's what we're working on right now. Now that's a hard problem.

Absolutely.

You know, I look at some of the products I use every day and I'm like, I don't have to go on a course, I don't have to watch YouTube videos. Nine times out of 10, I just go in and use it. So how do we do that for data management? And you know, we haven't solved that problem yet, but we're spending a hell of a lot of time trying.

Absolutely. And so in your experience of building this platform and working with your customers and helping to make that challenge of analytical use cases tractable without having to have an entire data team to be able to support those end users, what are some of the most interesting or innovative or unexpected ways that you've seen the agile data platform used?

So for us right now, it's been the use cases that we've delivered. So I used to have a joke in my previous world that the two projects you never wanted to do was data migration and payroll. Because what would happen is if you were successful, it just worked. There was no hurrah. It was like, yeah, the data was migrated and it all turned up. Yeah, the payroll got turned on, everybody got paid. And so success was like, yeah, and everything else was worse. Data didn't turn up, we lost it. Payroll didn't pay you right. And people really cared. One of our first projects was data migration Project right was the customer was moving from their legacy on prem system to a new Cloudy thing. They had a vendor partner that was implementing their new Cloudy software as a service. That vendor knew how hard it was to do data migration, not because they didn't understand their problem, but because they didn't understand the core bespoke system that every customer had.

Right.

And they had to do all that hard work. They knew that it was a money sink. And then also it distracted the implementation. People who just wanted to focus on new ways of working get the benefit out of the new system system. So what we did was we became the middle person. So if you think about core business process, who does what? Customer orders product, customer pays for order within an organization that never changes unless they fundamentally change and pivot themselves as a business, their systems change. So what we did was we took bespoke data, we brought it in, we mapped out their core business processes. Customer orders product, customer makes payment. We then said to the new vendor, okay, are those processes Fundamentally changing. No, lots of admin change, status change, workflow change. But those cool processes exist. Cool. So here they are. How do you want the data we like to consume from API? Cool. Would it be good if we gave you the APIs that match your import? You know, your schema for importing migrated data? Yeah, because then we don't have to map it. Cool. We expose that data via a set of APIs that they hit. Now, really interesting. We had a contract in place with them which says whenever you find a problem, you'll tell us about it. We'll reiterate either our rules for the core business processes or the APIs. Right? So you tell us that something's not right, we'll go away and fix it really quickly. You go and hit it again and tell us we got it right. Naturally, what people do is they want to do it themselves. So they found some problems, so they built some cleanup scripts on their side and didn't tell us. And then as we were reconciling, things weren't reconciling. We're like, no, no. We've reconciled from source to that consumer API. Right. We know the data matches, but when we go into the. Into the app. App, the last software as a service, the data's skewed, right. We've lost records. And then we found out they were cleaning it up, and they were cleaning it up wrong. So we just said, right, we're going to go and solve that problem of why you needed to clean it. We'll push it back to us. You just hit the API. And by doing that, we saved a whole lot of problems around reconciliation and conversations because we knew where it was breaking and we can go fix it. So never going to do a data migration. One of the first things we did was that the other one is this idea of a data clean room, right? This idea of being data Switzerland. So customer had 38 different people that needed to give them data. Some of those organizations had data teams, some didn't. So we needed to combine it and make it safe so that our customer could only see a subset of the information they were allowed to see. So some of the larger organizations would filter and clean the data before they sent it to us. So we only saw the data our customer was allowed to see. For some of the other providers, though, it was easier for them just to give us all the data and for us to apply the rules. So that's what we did. Right? They give us every event and then we cut it down. And then our customer can only see the stuff they're allowed to see. And then you go, okay, now what happens? Well, this is kind of event viewing use case. So, you know, some of the providers are using Google Tag Manager, some are using Google Analytics, some are using Adobe. Then we had to go into the Insta, Facebook, Twitter. Then we had to go into podcast events, then we had to go into connected TV events. And if you want to see complexity, go and talk to people that actually have video on demand on TVs and see the problems they have. Because what I didn't realize was every TV provider, Samsung, Sony, uses a different SDK for tracking the app usage on their TVs, and they often use different SDKs for different models of TVs. So if you think about the complexity of just capturing who viewed what, what, when you're reliant on the device provider, not your app yourself, you're not in control. There's a whole lot of complexity there. So, yeah, that data clean room. Right. Was going to be simple, but yeah, massive amount of complexity every time we touched a new set of data, which comes back to my problem, that data collection is not a solved problem. I thought it was. Right. I thought 5 train and such data had rocked it and solved it for us. Yeah. In my experience, every time you get a new data source, there's a new problem that's going to turn up and bite you in the back.

Absolutely.

And in your experience of bootstrapping this platform and building it out and exploring this overall space of building a data analytics backend as a service, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?

That often we do things as humans without thinking about it and it looks simple and when we try and codify it into a platform, it becomes hard. So in the previous podcast I talked about that idea of key identification, understanding when a table comes in or a piece of data comes in, where the unique key is to say, that's a concept. There's one we're working on right now which is really interesting. So it's around this idea again of removing clicks. So as I said, when I do something for a customer in our product and I have to click a few times or it takes me a while, we want to automate that, where I have to think and make a conscious decision. We want the system to make a recommendation on how to deal with that. And so as I said, we're a three tier architecture. Right. Data comes into the sisterized layer, we then model it lightly. Concepts, details and events and then we make it consumable. And so when you create a concept in detail, you've got a concept of customer and some detail about them. We automagically create that consume table, that big denormalized table on demand, right? You don't have to do anything. It just brings back the detail and concept and denormalizes it out and makes it available for query. When you create an event customer orders product, we automatically go and pick up the customer, the product, the orders, all their details and denormalize that out to consumable event table where you can see every order, every customer and every product that was ordered at that time. And that's all automatic. I don't have to do anything for that, but I still have to do the rules in the middle. I still have to say for this historical data, that's a customer, that's their name. And that feedback loop loop was a little bit time consuming because I get some data coming in and I can profile it in the history layer. So within our catalog you can profile it and see the shape of the data and nulls in there. But I naturally wanted to go and just query it, right? I wanted to go and do some draggy droppy stuff sometimes to get a little bit more context around it. And we didn't want to build that in the product. So what we're just finishing off now is when you land a piece of data, you have a choice for auto rule. And what it will do is it'll go and grab that data, we'll have a guess of what the key is and ask you whether it is or not. And if it is, you go yes. And then it will build out all the rules in the design stage and create that consumable table for you. Now we're not a great fan of source specific data models. We think you should always model your business processes, not the way your source system works. And so the idea of conforming concepts, you know, single view of customer, single view of product across multiple systems we think is highly value valuable. But doing that big design up front actually sometimes doesn't have value, right? You want to do it later. So this idea of auto rules almost gives us a prototyping capability. We think you land the data, it's still going to lightly model it for you. You're slightly designed, you then make it consumable and then you're going to use it for a while and then you're going to go back and refactor. And we think that prototype process, drop it, use it it's still lightly designed, there's still rules being created so it's not ad hoc. And then you're going to go and refactor your data design in the future. We think that life cycle may be one that has value, but as with all things Agile, right, build it quickly, get it out, use it in anger, see what happens. If we stop using it, it didn't have any value, and then we should burn it down, which is hard keying.

Off of that doing it Agile statement. One of the hallmarks of agile is that you do things quickly and you iterate rapidly so that you can identify the mistakes early rather than waiting until they become more expensive. And I'm curious, what are some of the most interesting or informative mistakes that you've made in the process of building Agile data and some of the useful lessons that you've learned in the process.

So there's been some that we planned that we knew we had some technical debt. So last time I talked about the idea of our config Being in BigQuery, we knew that we had to iterate that that we did the no SQL move to data store because we thought we wanted to be cool. And we ended up throwing all that work away and refactoring it into Spanner, which is now a foundational piece for us. And we won't change that unless we really had to. So that was kind of an investment mistake where we knew we had to make a change and we made a wrong bet and we lost that work and that was okay. There's mistakes in the product, there's things we've built that I don't think we ever use. So when we built out the data catalog, I really wanted to review thumb up, thumb down, done right. I thought it would be valuable. We profile the data with those statistics are available. We have context that you can add against the data about the history of, you know, where the data came from, why I should care, any gotchas? But that review process of, you know, the Yelp voting, yes, it's good and no it's not. We built that out, didn't take us long. I'm not sure it's ever going to get used. I mean, we monitor it, right? So we'll see if that never gets used. We'll probably take it off the screen. Screen. The next one is we build out a whole lot of what we call trust rules. So they are data quality rules or data test rules. So when you're building out your change rules, you know, make this into a concept or detail. A screen pops up and says here's some rules you can apply. You know, is it not null? Is it unique, Is it a phone number? Those, those kind of simple data quality tests and that information is all displayed on the catalog. Yeah. So you can go into a tile and you can see all the rules that ran the trust rules and whether they passed or whether they failed, built. So that's been successful from that point of view, but it's not actionable. Right. We haven't built the but we actually needed was what happens when it fails. So that's what we're going to do now. Right now, whether refactor the trust rules or not, we don't know. But we have to solve that next problem. Right. Okay. Something failed, do I really want to have to go and check it? Do I want to be notified of it? If I'm notified of it and I get 600 failures. Right. For different things, is there a classification of these ones are important, these ones aren't, should I be able to mute them? What's the actual action I want to take? And so that's one of the lessons we've learned. And it's a product lesson as much as an agile or a data lesson is actually when this feature turns up, what action do we expect an analyst to take by that feature? What's the value in it? And it took me a while to flip the model to think about that first. Yeah, but having said that, there's a bunch of vanity features that I really want and it's really hard to beat yourself up and say, no, we're not going to build them next next because we probably don't have value for them or what's the least we can spend on building it? And how do we prove it had value or it didn't? And if it doesn't have value, you know, that is a hard human process I've found because I don't come from a product background traditionally. Right. So it's all been a good learning for me.

Data engineers don't enjoy writing, maintaining and modifying ETL pipelines all day, every day. Especially once they realized that 90% of all major data sources like Google Analytics, Salesforce, AdWords, Facebook and spreadsheets are already available as Plug and Play connectors with reliable, intuitive SaaS solutions. Hevo Data is a highly reliable and intuitive data pipeline platform used by data engineers from over 40 countries to set up and run low latency ELT pipelines with zero maintenance, boasting more than 150 out of the box. Connectors that can be set up in minutes. Hevo also allows you to monitor and control your pipeline. You get real time data flow visibility with failsafe mechanisms and alerts if anything breaks. Preload transformations and auto schema mapping precisely control how data lands in your destination, models and workflows to transform data for analytics and reverse ETL capability to move the transformed data back to your business software to inspire timely action. All of this, plus its transparent pricing and 24.7live support makes it consistently voted by users as the leader in the data pipeline category on replacement review platforms like G2. Go to dataengineeringpodcast.com hevodata today and sign up for a free 14 day trial that also comes with 24. 7 support.

For people who are interested in being able to reduce the overhead involved in being able to just ask questions about the data for their business without having to invest in building out a whole team to support it. What are the cases where Agile data is the wrong choice?

So for us it's real time. We're definitely not a platform for real time reporting. Yeah, we can collect data in real time because effectively we're running on Google Cloud. We can use the pub sub queuing topic stuff where people can actually stream us records in real time and we can just land and store them into the history layer. We could in theory take our rules, which run sequentially at the moment. So if you think about it, I think about it as London Underground data comes into a history by default it turns on what we call auto sync. So as soon as a new record comes up into a history tile, we automatically trigger every dependent rule and it uses a manifest process. It kind of dynamically builds the DAG every time rather than have the DAG stored. So we query the config, we go right, what's dependent on the history tile? Run those rules. Okay, what's dependent on the output of those rules? Write these things, run the next rules and we daisy chain through that manifestation. So yeah, that runs pretty quick because Google just scales like snot. But it's still sequential, right? It still waits. Now in theory we could pick up the entire config, encapsulate that end to end piece of code as a single piece of code that runs in memory. And every time a new record comes up in real time, effectively re instantiate that code and give a real time score or answer at the end. But we're not going to invest in that. It's not a big use case for us. It's a high level of complexity for engineering to get that right. And we find very few use cases where a customer actually wants their number to change in real time. At the end of the mile, where I look at a number and it's just changing. And so that's not a use case for us. Right. We could build it, but we're not going to. You know, that's probably the main one. Everything else is we are really good at it or we're going to get good at it, as long as it's a data problem, I suppose the other one is if you don't want the platform to store the data, if that's something that you can't let happen, then we're not a product for you because we do collect the data and store it in history. It's a foundational piece of our pattern. We've got Google Omni, which is really interesting, which in theory says you can have your data in S3 or Azure blob storage and we can go and hit that. But we'd have to change a bunch of patterns to say that you hold that history layer, which we can do. But what happens if you change it? What happens if you delete it? Right. We lose that immutable record. So we can't replay all our rules and give you the same answer anymore. And for us that's an important feature, right? Is that at any time you can see what the number you reported was. And we think that's a core part of data for what we do.

As you continue to build out and iterate on the product and the platform, what are some of the things you have planned for the near to medium term or any particular projects or problem areas that you're excited to dig into?

So we kind of look at in two ways. Every time I touch something and it takes too long or it's complex, we're going to go and make that puppy simple. And then there's a bunch of core capabilities or features that we know we haven't invested in yet. So, you know, if I look at the whole idea of semantic metrics layer, we've done the designs for that and we're waiting for the kind of looker email service to come out so we can bind our API and our app with that service layer so we don't have to build out the execution of those metrics ourselves, but we've got to build that out. And so at the moment, when I build a metric for a customer, I'm nine times out of ten, I'm building on that last mile tool and we want to move that metric definition back into our product, almost that headless BI pattern, because we see that has a high value. We want the machine to do more work. So we want to get more into writing algorithms under the covers that provide recommendations. So we have some simple ones here at the moment. Things like data drift, where we run algorithms to tell us that the data looks like it's skewed and it's going to alert us, hey, that looks weird. Go look at it. Rather than waiting for the customer to say, hey, the data is kind of not what we thought it was. We want to build more of that. So more of those models, but we see them as recommendations. They're not doing the work. They're just the machines giving us a hint that to reduce the work that we do as humans. And the last one that I'd really like to do, but it's one of the ones that Google removed, was the idea of natural language Q and A. So we do research spikes every now and again. We call them mixed spikies. And I was really keen on this idea a couple years ago of while we're not going to build a last mile tool in our product, nine times out of 10 people just want to answer a data question, how many customers have I got? What products did a customer order? What was the total sales in this region? And so that whole natural language ask a question and get an answer. So we did a McSpikey to see what it would take for us to build that out. And we estimated if we found five really, really smart engineers in that space and we gave them a year, we'd have a really basic but crap capability to ask a question and get an answer. And we're like, okay, we're not going to invest in that. And then Google announced their Q and A service as private beta. So we jumped on that one and what that was was the ability to ask in SQL a natural language question against BigQuery and get an answer. And we tested it out and it was a thing of beauty. It was like, wow, that's actually better than a lot of the commercial products we've seen. And then we were like, why is it good? And it was like, well, I didn't realize that Google Analytics, you can go and ask it a question and it's a natural language question. And Google Sheets, you can go and ask a natural language question. And Google Slides when it's recommending some style changes. That's a natural language question. So Google had taken the whole dictionary capability out of Google search and Been testing these dictionaries and these language algorithms and these other products of these for years. And then they embedded this in the Q and A service. So we're like, wow, that's cool. In our product we have a thing called eighty, a little chatbot that comes out from the side and then when you want to do something it says, okay, you've done this. The next step we normally would see is do this next. Or okay, you're going to do this, you've got a choice, right? For example, if you're going to go delete a rule and there's a draft rule in play, do you just want to delete the draft or do you want to delete the draft and all the previous versions so effectively remove that rule completely? And it's no longer in production. So AD pops up and says, which of these two would you like to do? So we go, cool, really simple. We embed natural language into ad. We have ask Adie. You ask her a question, she'll give you an answer, right? Because the config holds all the metadata, we understand what each field means, we understand the how it's used, we understand the rules. So that was all cool. So we backlogged that until we had some time and we made it a priority. Come in six months, 12 months later. I'm like, cool, let's go into building a cool feature because we just need a bit of a break from the plumbing stuff. Go back to go and touch it and it's gone. And it's like, okay, reach out to the product team. Where's that gone? Oh, you know, we've turned the beater off. Okay, so when's it going to come? Oh, it's now not. You might see it in the BI tool, in the looker, Google Data Studio stuff is where we think it will turn up, but it's not going to be a core service we can use. And that was like ah, that saved us years of engineering. I mean we're not going to engineer it ourselves, right? But it was like ah, I really wanted that. I think it had high value. So ups and downs, you know, that's what happens when you are relying on other people's technology often to do stuff so you don't to have to. That's okay.

Yeah, especially when that someone is Google who has a long track record of releasing really cool things that people like and then saying, nah, we changed our mind.

That's the definition of agile products, right, is you go and invest in something and then you build it out. And if People don't use it, then you kill it. That's hard, right? Because somebody's using it. Not enough, but you know, somebody's relying on it. But that's got to go. So yeah, I mean they are the epitome of agile productivity development, but as a consumer we get to love to hate that behavior sometimes.

Absolutely. Are there any other aspects of the work that you're doing on the agile data platform or the overall space of building a backend service for analytics that we didn't discuss yet that you'd like to cover before we close out the show?

I've focused a lot about our product and how we have back end services and front end apps that are just beautiful to help the industry that works work. But I've probably underpaid a lot around that way of working that. Actually you still have to be data savvy. And I use the word data savvy now on purpose because I read an article or listened to a blogcast or somebody had this really good comment which is when you tell somebody they're not literate, actually you've been quite derogatory. Everybody's literate. It's about the level of literacy. And so as engineers and as data modelers, there's a big thing going around LinkedIn at the moment around data modelers and the fact that we've lost the art of modeling. And for those of us that believe modeling has value, we see the loss of it as a big loss and we're quite negative against it. We need to use words that are polite and true, right? So you don't have to be an expert in data, you just have to be savvy, right? And so you need to practices or ways of working that help you understand how to take the savviness that you've got and apply it in the data domain. So that idea of data modeling, right, the idea of saying, well, there's only three things you need to worry about. You have a concept of a thing, customer order, product. You have some detail about it. Customer has a name, product has a SKU or a name or a type. Order has a quantity and a value and a date. They go together in an event which, where we see a relationship, we see a customer order, a product and we want to record the fact that they happened because it's important to us. If you take that way of working and you combine that with a product so that they have that ensemble, that's where we think the value is. That's where we think we can enable the analyst to do the work that an engineer. The plumbing work. Right. And I'll go back to that. The boring old pipes that just move the water. Yeah. We still need smart people that are really good at problem solving to do that hardcore engineering analysis, problem solving at the end of that pipe. Right. That's where the value is and that's where those constrained resources, those people that don't have enough time should be focused. We should just automate the plan.

Plumbing.

Right. It's what we do in our houses. Why don't we do that with our data?

All right, well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.

Same answer as the last podcast. Right. I think there's too much complexity in our platforms and the way we work and that's what we're hyper focused on for clients. I still think there is a lack of machines doing the work for us where they can. The example I used last time was finding the unique key on the table. Actually, somebody on LinkedIn replied back and said the data quality tools of 20 years ago did it. They found the foreign key relationships. And yeah, that's true, but I don't see it in the modern world again. So we've kind of lost that art. So for me, it's when the machines get smart enough to do the work for us, us and we don't notice, we just take the recommendation and go. Yeah, that made sense. And that's where the data world needs to get to. So hopefully that's the next generation of data platforms, ones that make our toast for us, brew our coffee for us. Right. That kind of thing.

Yeah. And do it at the right time so it's not cold by the time you get to it.

Yeah. You know, just think about it. The Nespresso. Right. We still have to put the water in, we still have to put the pot in, but that's it. Yeah, we still pick the size of coffee. We want the strength. We still have a lot of choice. But the core plumbing, that's taken care of. Well, it's not really, is it? Cuz I still have to fill up the water thing. But anyway, you know what I mean?

Yeah. All right, well, thank you very much for taking the time today to join me and share the work that you're doing on the Agile data platform and product. It's definitely a very interesting approach that you've taken, an interesting service that you're providing. So I appreciate all of the time and energy that you and your co founder are putting into making analytics a set and forget operation as much as possible. So thank you again for that and I hope you enjoy the rest of your day.

Yeah, thanks for having me on. It's been great. As always.

Thank you for listening. Don't forget to check out our other shows Podcast Init, which covers the Python language, its community and the innovative ways it is being used, and the Machine Learning Podcast, which helps you go from idea to production with machine learning. Visit the site@dataengineeringpodcast.com to subscribe to the show. Sign up for the mailing list and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hostsatangineeringpodcast.com with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and co workers.