AgileData #42 - The Activity Schema data modeling pattern with Ahmed Elsamadisi

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Ahmed Elsamidisi.

Hey Ahmed, thanks for coming on the show. Today we're going to go into another one around data modeling, this time activity schema and I'm mere excited because I've read a bit about activity schema, had a little bit of a play with it but nowhere been in the weeds. I'm looking forward to understanding what the pattern is, when we should use it, more importantly when we shouldn't and diving into it. But before we do that, why don't we get a bit of a background from the audience about you, how you got into this lovely world of data.

Awesome. So I actually started my career in self driving cars in 2010 before it was cool and so got to spend a lot of time focusing on AI and decision making. Moved my way into AI for missile defense for the US government and then eventually joined the startup WeWork and built out our data team and watched it scale. So I got to see data used in all sorts of different ways from mission critical applications to business decisions. And what I found was that having implemented the system at WeWork, we went through maybe four or five data stack implementations. We kept running into issues and, and couldn't find any good solution. So what we realized was that everything was always coming down to data modeling and data modeling always is the problem for why data scales terribly. And I think a lot of people nowadays are beginning to know that with DBT publishing an article about how their data modeling didn't scale and so many things of that nature. So I'm excited to show this paradigm that we've been using a lot of our customers use and it's, it scales really nicely and hopefully more of the world can experience it.

We know that bringing back data modeling is kind of one of the themes this year and make data modeling cool again. For me, you've come from working on self driving cars and working on rockets to data modeling. How cool is that? Right? You've gone from being a rocket scientist to a data modeler. That's a progression. Before we jump into activity schema though, let's just because we tend to go all over the place on the podcast and that's all pretty cool. We always come back to the core thing. But that idea that at WeWork you went through five data stack changes, talk me through that. Was that because there was new cooler technology out Was it because the first four didn't work? Was it because you're just iterating? Because that's a fairly expensive change, depending on what you mean by changing your data stack.

Yeah. So I feel like a lot of people actually go through the same flows. We had a postgres database Tron Scheduler that would schedule our queries and materialize them. And we had Chardio for visualization. And we're like, this seems great. And then we just had so many things scheduled. Answering questions was really hard and the company was like 200 people then. So it wasn't really that much. That many questions. And we're like, aha, I'm gonna upgrade my database. I'm gonna switch from Postgres Redshift from Unicron to. At that time, Luigi was popular. So we used Luigi and then we had Chardio. And still more and more data models still can answer questions. And we're like, ah, I know what it is. It's not Chardio. We need Tableau. So we got Chardio and T. And we're like, that's going to enable us to answer more questions. It didn't. So we're like, okay, probably instead of Luigi, let's switch to Airflow. And so we had Redshift Airflow. And instead of Chardio, we switched to Looker. So we had Looker and Tableau. And we're like, that was going to help. And then we're like, people are still yelling at us and numbers are not matching. So we added Anadot. And we're like, that's going to solve the problem. We added a dictionary, we added some lineage. Still numbers don't. Like numbers still don't add up. Sometimes people have so many questions, just like the backlog, we're like, aha, it must be Snowflake. So let's switch from right here to Snowflake. And then we're like, fuck Airflow. Let's build our own custom scheduler that does our own things. So we built something called We Module and Data Manager. Oh my God. And then we had Looker, Tableau, Chartio. And then we had another tool somebody bought Wave analytics and we had. It's just so many tools. Like it was just so many tools. And we had a 45 person team and we still had the same problems. People would have questions we couldn't answer. Oh, we also switched from. We also bought Heap and Amplitude to help. And like, still people had so many questions they couldn't answer. Everyone was complaining. A lot of the dashboards didn't add up. And it was just a shit show. So we're like, why is that the case? So we actually went on a tour. And this is an interesting thing is like, I went on a tour, I talked to a bunch of big companies like Airbnbs and Netflix and Spotify, and I was like, how do you guys deal with this? And it turns out every company is in the same boat. Everyone is just refactoring the same things. And every time you refactor, it seems great for the first couple months, you're like, oh, my God, I just threw away. And then you refactor or you buy a new tool and you just move all the stuff that's already working. So every tool appears incredible for the first couple months because it doesn't have all the shit yet. So that was exciting. So all these companies were just like, yeah, that's the job. The job of data engineering is to manage the shit show. That's it. There's no solution. You just try to minimize it. And all these tools are helping you manage this growing chaos problem. And what I wanted to do is, is there a system that's designed for why does chaos of data continue to grow? Understanding what causes it and then understanding what part of this whole data stack is the real problem? Because as you dive into it, you realize that Snowflake and Redshift and we bigquery, like, the slight differences don't really matter. People love talking about it as a millisecond faster. I don't care. I have these things being materialized. And then people talk about the query editor and whether it's sync to GitHub embeds. Jinja. It's like, okay, but, yeah, but it doesn't really matter. I still am running compiling down to SQL. So, okay, so I've used all these different tools that haven't really changed our approach. I use Dashboards and Looker. Whether it's purple or Chardy or Tableau, whatever color their color scheme is doesn't really make the solution better. So we're just constantly using new tools and using those tools as an excuse to refactor. But this fundamental approach doesn't work for at least what I've seen. I just couldn't find a single person who was like, yep, a year into my stack and it's great. Everyone's like, a year in and they're like, I want to refactor and blow this thing up because it sucks. So that's where my whole life was like, okay, I need to really dive into this problem and let's go figure it out.

Yeah, it's really interesting, isn't it? It's interesting. As data people and as technologists, we love to refactor our stack. We love to refactor the tools, we love to invent stuff not invented here, but we hardly ever refactor our way of working. We sit back and we watch our data flows, we watch the data move and we go, oh, there's a bottleneck over there. We're going to go. And we're going to go and refactor that bottleneck. But we hardly ever sit back and watch the way our team works. We don't treat humans as part of the system and go, where's the bottleneck? Where's the bleed? I was lucky enough to work with a guy called Sean McGurr many years ago and my consulting company, and he now works for Data Iq. And he was running. Before that, he was running a analytics team over in the uk and he told me the story. He went to a conference in the uk, meet up, stand up, whatever. On the stage was one of the fan companies, can't remember which one again. Airbnb, Amazon, one of those big companies. And he said to them, he said, how do you deal with the data problem? And the person on the stage says, what do you mean? And he goes, when you want to find some data, you've got so much of it, how do you find it? The data scientist said, what's easy? I just go into Slack, I log a call, 15 minutes later, the data team have got me the data. And so they had hundreds of people that served the data to the person that needed it. And they scaled with bodies, not with ways of working and not with systems. So for me, that's fine if that's your business model and you can afford it, but for, for most of us, we can't. And it's not overly efficient. And if you think about that, five changes stacks. At what stage did you bring in data modeling?

First of all, the story you had is such an incredible highlighting of the problem, which is even the greatest. The most advanced tech companies that sell you the software to do all these things, they just shove people at the problem. It's their whatever. I'll just have people go deal with it. So that's just kind of an interesting insight thing. But the question I think is always going to be, why? Why is it hard to find data? I have 1,000 tables, so which table do I use? Why do you have 1,000 tables? The idea of the activity schema, which I'll bring it to. That is what if you had less? So like if you had one table only, you would say where is the data in that one table? Problem solved. Cool. If your data was standardized in the structure. If you were like, what data is in the table? The structure is consistent so you don't have to ever. What does this column mean? Once you have one table, then data lineage goes away. The lineage one data dictionary goes away. Because the table should be self explanatory, it's consistent. Instead of going and messaging someone to be like, where do I find data? It's I only have one place to look. So having the concept of one table is really kind of nice because it makes things so much simpler for processing, for understanding, for finding, for the way that humans actually deal with data. Now the question is why doesn't everyone just build one table? That seems like a simple solution. And I think to understand that we have to go back to.

Why do we build multiple tables and why do we build them more? Interestingly, why do we build it the way that we currently do? And that was like my obsession. So the reason why you build multiple tables is you have different questions. So you're like, okay, cool. Every question has, you have to answer the questions and therefore you need data. But you might think, why don't I have one table for every concept that I have? Because the company doesn't have many concepts. And then all I have to do is combine those tables and I'm problem solved. And I think there lies the problem combining those concepts into tables that are used to answer questions. That's what like 95% of your tables are the combining and with questions and the way people ask questions. There's constantly new questions that relates data differently. And because you're constantly asking questions that are unique, you're constantly relating data in a unique way, which means you constantly have to build a new table to represent that data.

So the follow up question is, okay, then doesn't SQL allow me to do that? And then so this is where the data splits off. So the star schema was designed in a world where you had one data system. When it was designed, you had, every company had one postgres data storage was very expensive. Compute was also expensive. You would take your data from one database and build aggregate tables so you can actually visualize X and Y because you're working within one system. You had foreign keys. If anyone who's listening to this, a foreign key is just a way to join data with itself. And the questions people were asking was how Many comments are on my ticket. Therefore you join comments and tickets, aggregate it, and it makes perfect sense. Today's age, every single tool has embedded analytics. No one's asking your data team how many comments are on a ticket. Like Zendesk or the Ticket tool will give you that they're asking more behavioral questions, which is okay. When people submit a ticket, are they more likely to churn? Or what part of the app resulted to people submitting the most tickets? Shit. Now I have two separate systems and those systems don't talk to each other. So you need to write some sort of complex query to combine the data in a way that answers that question. This is a separation between writing what people talk about. I know SQL, it's like, sure, and every single BI tool can generate SQL. But the SQL to combine those two sources is very complicated, it's very nuanced, it's never predefined. And that's what your data team is spending all the time doing is combining that. Because it's so hard to combine it and requires so much SQL and code, you end up saving that table. And that's why you end up going to a person who, who knows exactly what makes that table. And then they can tell you, here's a table to use. That's where we are today. Does that make any kind of sense at a high level?

Yeah, look, there's so many patterns in there that I just want to unpack. So where do I start? Where do I start? Okay, so let's, let's go to the star schema one because really it intrigues me that the US is the king of star schemas. And actually we see Europe as being more data vault modeling. And what's intrigued me is when DBT started out and they did that awful thing because I'm hyperfocused on terminology. Language is important. If you see a cat and you call it a duck, and you keep calling it a duck, it's still a freaking cat. It's never a duck, no matter what you do. So the fact that they came out and they had this term model which was a blob of code, it's done so much damage in the marketplace, in the data world. And then when we start moving towards self service democratization, we know that we lose rigor and that's okay. We're getting a whole lot of new people who don't know how to do it in the way that we were trained. And therefore we got to help them apply this balance of self service and rigor without making it awful for them. So what we see in DBT world is they've now realized that 5,000 blobs of code is probably a bad thing. We could have told you that. But that's okay. You have speed of market on your first hundred and that's good. So what do we do? We go, oh, you need to model. And for some reason everybody said dimensional modeling is the way to go. But if we look at dimensionally model and star schemas, what was it there for? Was there because our technology was so constrained we couldn't just dump all the data in one database and ask a question. We had to make choices of how to optimize it. And the star schemas was a brilliant way of optimizing it. People keep telling me, oh, analysts really love star schemas and understand it. And I call, yes, we can teach them that. A dim and effect and the grain of the fact and you join back to the dim and then you've got to do all these horrible joins. But actually, in my view, you go ask an analyst what they want. One big table, just give me a table of the data. And they go, oh, if it's customer orders, you can't count customer. You go count distinct customer and you get the answer. Oh, they won't know that. And then I argue sold them. You're telling me that actually you can teach them how to join facts and STD, type 2 dimensions and get the right answer and actually somehow count distinct when you've got a table that's got multiple orders for a customer is more challenging. Call bullshit again. The next thing is we know the first question we always get is a simple one. The first question we ever get is how many? And we know that's not actually the question. That is an exploratory process where somebody says, give me the answer to that and then I'll know what the next question to answer is. And it's typically, how many or how much? Yeah, how many orders do we have? Or what was the total order value? Then we have some kind of breakdown. Where did it happen? Which store, which product type, which order values above 100. So we get these wears and these buys and then we're going to get the how longs. Oh, how long was it between them placing the order and our shipping? How long was it between them placing the order and paying? And then the exceptions, oh, how many people placed an order and paid and we didn't ship? Actually, how many people placed the order and we shipped and they didn't pay because that should never happen, but oh, it happens in the data. And then we get to the last one, which is why? Why did they place the order and not pay for it? Yeah. And so that story of answering those questions is repeatable. We see it time and time again because the person doesn't actually know what the real question they want to ask is yet they're using data to explore the problem, get that question and then they want to answer it. So yeah, I reinforce the messaging you're giving. Right. Is that it's a problem for a long time.

Well, I want to add two more things that happens after in between. The why, which is key is that part of the why is they have hypotheses and the stakeholder hypothesizes doesn't care about the order. So they say, did that person receive an email campaign? What was the last campaign they received? Did they come to our website? Maybe they submitted a ticket between when they got shipped and I want to know what happened there. We all can agree that's the flow. Now when you look at how we answer those questions, we treat every step like a production system. So if we were going to write this whole query to do it, well, now you want to add another concept to that query. Oh my God, that is super complicated. Because now you're combining multiple and multiple systems, that query gets really long. And because the way that SQL is stack based, which means that SQL is joined, any sort of thing you're not careful with, you can duplicate rows, you can drop rows and you can cascade and blow up your whole thing. So I think that process of how we answer that should be the way that we're talking about it. I ask that question, I answer it, I ask that question, I answer it. And it should feel like that kind of minutes. But today we have to go through a whole fucking cycle, 20 questions that we know we're going to ask in that order. It would take a data team two weeks per question in a whole cycle and materializing table and I have to maintain 50 tables and that shit show. So that's one thing. The second thing that I want to touch upon that you also mentioned is humans and words. Words are so important. We used to have this joke which is people say, okay, how many sales that I get? Well, it turns out that if you asked the sales team how many sales they get, they care about the timestamp being when the customer signed the contract. If you ask the entire executive team, they care about when the customer started their reservation. If you ask the finance team if they care about when the customer made their first payments. These things are different. So just think how many sales did we get is really unhelpful. And what we have done is we've expected so much information to exist in a column, in three words of a column. So we see this table name sales and web data, and you're like, cool. And it has a column total sales. But you don't know what that total sales actually really means because that total sales can be based on the first touch, attribution based on the last touch, based on when the sale happened. There could be so many things. And that just language issue is such a fundamental issue. So that's the first language issue. Second language issue that happens is that the person who's asking a question is asking it exactly the way you did, which is, did they call? Okay, how long did it take them from the next order? They're describing it in a very human centric way. They're talking about building blocks or actions that the customer is doing and relating those actions. The data engineer or data person that has to answer it has to take that question from action and time and relating it in this language in English and convert it into SQL, which is Stack table and how do I join it?

That was the first insight that led to activity schema. It said, what if we, we don't have to change it. What if we can actually take a structure that represents the way people talk about data and what if we can actually relate data in the same way that people think about relating data? Those two aspects is what made the activity schema what it is. The first concept is that you break your world into activities, which is they completed the order, got shipped, they got a call, they submitted a ticket, they did this thing. The second thing is that you relate data to in the activity schema world using what's called temporal joins in between the order. Did they call us? Give me the last time before they called us what the ticket was. Notice how in those worlds you never predefined any joins, you never have to worry about which system it came from. And it's super analogous to the way that humans are asking questions. And by creating a structure where you can combine data that way and ask and answer questions that way, it just mitigates that cycle. So you don't have to worry about the foreign key not existing, you don't have to worry about weird joins, you don't have to worry about duplicating rows, you don't have to worry about dropping rows. You can add and Then when you want to add something, because activities can always be appended, you can add many temporal joins without ever changing the number of rows in the data. Once they go, okay, I added the in between, did they get a shift? Okay, did it get returned? Okay, wait, did before they submitted the order, did they submit a ticket? I can always add as many more dimensions as people ask follow up questions and slice and dice that data without ever having to worry about going back to the original definitions of models. As long as the concept is defined as an activity, you can just use it and be happy.

So let's unpack that a little bit. I'm going to apply it back to you the way I understand it. Tell me when I'm wrong. So you use the term activity, I could use the term event, or I could use the term core business process. Or in data vault terms, I could use the term link. Or in dimensional modeling, it's like effect. Again, the example I always use customer orders, product. Customer pays for order or do they pay for product? Good question. Store ships, order or do they ship a product? Good question. Customer returns product to be able to return the order. So those are four core business processes or four events that we'll see in the data relationships and concepts. And so for activity schema, what we're talking about is the term we use is activity. And we say we saw that activity happen. From my podcast with Hans, we talk about Peter the Fly. If I was sitting on a wall and I actually watched what happened with the humans, I see a customer and I see them order that product. And so what we do is we rack and stack those activities into a single table. So now we've only got one place to go. You want to ask a question, that's the table for you. One table to rule them all. And then the second thing is we know that often the question as we talked about is the why, what's the time between these activities happening? Is actually the hard question. And why is it hard? Normally in a dimensional world, we've got to go away and we've got to grab multiple fact tables, multiple grains, slam them together, and then be able to do the time series query about temporal when did that happen? When did that happen? What's the difference in time between those two? Or in data vault, we've got to grab the links, we've got to slam them together and we've got to say, I've got multiple events now or activities, and then the same kind of thing. So with activity schema, by putting the activities in a Single table and having some core metadata effectively around business effective dates. And when that thing happened, we can now just write a windowing function. It's the same windowing function to say, take this activity customer ordered product and take this activity customer made payment and tell me the time between, because we know the structure of the table and that query becomes repeatable because it's exactly the same pattern of query. Did I get that right?

So 95%. So the activity schema way is instead of thinking about it as window functions, we like to call them temporal joints. And we found there's only 10 of them. So why are we not using window functions? Because as you add data and you want to add columns, you want to be able to pull more nuanced relationships. The temporal joins that we have is either first ever, last ever. So give me everyone's order and give me the first web visit they came from and so I can see the ad source from it. Easy. Give me the last order they have so I can know if they're an active customer or not. So that's first ever, last ever. We also have first before, last, before last. Most common when they have this order, give me the last time before and you can add within 30 minutes. Did they submit a ticket? Then there's in between, which is the most powerful one. Okay, now I know in between orders, did they email? So not just not ever after, but in between the specific interweave the orders give me. Did it do it? So by having these very specific ways of actually doing it, you end up creating a more consistent, standardized way of relating data, which is these 10 temporal joins. And you can add as many more dimensions relating the data to the original, starting what we call the cohort activity. So you start from an order and you can add as many things last before this, first after this, in between this, aggregate all the orders over aggregate all ever. So I can have the LTV all the sorts of things the next 30 days. So I can get LTV for the next 30 days. All that becomes really easy because you're always stacking that data or appending that data to it. So that's the. It just creates a more standard way of relating the data. And you're always trying to describe that customer journey. I care about when this happens and when this happens in between, or when this happens or this happens after, or that sort of thing.

Okay, so if I don't use the word window function, let me give you another pattern that we had years ago. So we used to have OLAP cubes back in the days when, again, I know it was constraint based, our relational databases couldn't hold high volume aggregate queries. So we then had another database which was multi dimensional. So roll it. Molap, holap, a whole lot of different techniques. And the goal of that was be able to get aggregate queries faster than relational databases. And we typically had analytical functions. So what you're saying is with the activity schema, we get 10 patterns, 10 query patterns, 10 analytical functions that are time based. So our ability to ask questions over time for those activities or events and the relationship between them gives us the answer.

Cool.

So did I get that right?

Yep, perfect.

Excellent.

Okay.

Activities are events. We rack and stack them into a table so we know that there's one place to go to see all the events that relate to something. We have these 10 query patterns, these 10 analytical functions, these 10 temporal things that say we can ask those 10 questions and they always will give us the right answer. And they're the top 10 that we always use. And they're quite right if you have to write them by hand. And then the last thing we want to talk about is concepts. And the way I describe it is customer, orders, product. I have a concept of customer, concept of order, concept of a product, customer pays for order. I now have a concept of a customer, concept of a payment, concept of an order. So you could think of concepts as a dim in star schema world, maybe typically it's something you want to count, it's something you want to manage. You manage your customers, you manage your suppliers, you manage your orders, you manage your employees. So in activity schema, where do the concepts live? Where do the things that bind that activity together to say that these things had a relationship at that point in time.

So that's another uniqueness about activity schema is you actually relate everything to the core actor or the entity that you care about. So you might have multiple entities and multiple activity schemas. So based on your business model, we are multi sided platform. We have a person stream and a company stream because a company does things like pay, makes payments and stuff, but a person might log into a portal. So for the E Comm example that you used, one of the things you said is you already described it in the language, right? So the customer might complete order, we might ship an order. We also might have another activity called ship product. We might have another activity called purchase product. The customer is doing these behavioral things that we call activities, and that's where everything lives. So everything that you will tie as long as it's related to the actor. A customer opens an email, a customer has their order delivered. Whatever you want to do, if someone is doing it, you keep it as activities. And the reason why we do that is explicitness. Instead of having concepts. And you might have payments and payments. It's like payments exist as an object. We don't like to think about objects in the activity scheme world. We like to think about customer behavior. So a customer makes a payment, a customer has a payment get canceled. A customer may submit a billing inquiry. And as long as you keep it that way, it matches very close to how people are asking and answering questions. And that's what makes it really simple, is that if someone has a new concept that they're asking questions about, you can make it an activity. It doesn't really matter. Activities can overlap a little bit. It's okay. You might have, like you said, signed contract, started subscription made invoice, and you can always ask questions using those words. You can also have some sort of metadata that says this payment ties to this specific invoice. But for the customer's perspective, you can say, give me every time the customer had an invoice and give me the first time after they had that invoice. Where then the metadata, you can do the metadata additional join where it is the same id, but for most things you don't need to worry about that because it ends up being very less common.

All right, let me just replay that back to you to make sure I got the pattern and to reinforce it. And then I'm going to go into some of the complex ones where I think activity schema's got some problems, not problems where the context doesn't fit. Be clear here. Every modeling technique we have works really well in certain contexts. And these other contexts, we, we are the cost. So, yeah, with Data Vault, it's the complexity of the tables that we create, because like you said, we. We create a shit ton of them. And if you have to query them in this raw form as an analyst, you're screwed. We have to do some extra work to solve that problem. But let's go back to the core of that. So what we're saying is that the activity schema is bound by a core concept. So we might say we have a activity schema. Do you call them tables or do you like.

Yeah, we can call them activity streams. So your activity schema, you have multiple streams.

Okay. But effectively in my head, it's a table.

So we have single table. Yeah.

Cool.

All right.

And so we have one that's bound by the core concept of customer. So every activity that relates or every event that relates to a customer is in that table. Now, what's interesting about that is we have to alter our language slightly when we talk about who does what, right? So we go, customer orders product, Customer pays for order. That's good because we start off with the word customer. It's an activity that customer is doing. So we go, naturally, customer did that. But when I go to store ships product, it's normally store ships product to customer. So in a natural language, customer's at the end. But actually an activity schema, what we say is customer has product shipped to them from the store because we're binding it back to that core concept. So our language is. It's the same language, but it's slightly changed. We always put customer in the front. And if we can do that customer returns product, then we have a boundary for the activity, the activity schema.

So just add one more note on that's really critical. Makes asking and answering the right question really easy. Can I give one example that I like to give? I had a customer. They have an online platform and they were trying to understand when they launch a class, how many people sign up. So they want to know what's the best time to launch a class. They had done this analysis and they were great. We have a table. Its classes get launched and then they get whatever the average number of signups and they say, based on when the class gets launched, what's the average number of signups? Makes sense. So in the activity schema, that doesn't let you do that. It's because, like, you can't. You. What do you mean? When a class gets launched, it's. But that's important because when a class gets launched cannot impact the people signing up. Because if it gets launched at 6am and everyone gets an email at 8am you care about when people actually see the class, Even though you're saying, based on when I launch the class, how does it impact things? What you're actually saying is, when a customer first sees a class, how does that impact the likelihood for them to sign up for it? So based on when customers see classes, and that's when having that core entity makes asking those questions really nice. Because we saw a lot of bad decisions where people were. This whole story why this customer ended up buying. Narrators. After we did this analysis, we realized that, oh, it turns out when we launched a class had no impact. Even though it looked like it. It's because it's actually Based on when we send the email. Because that's when everyone actually sees the class. In the activity schema world, that would have never happened. They would never been able to make that mistake. Because you only tie things together through the customer. And therefore it's when the customer sees the class, not when a database updates something. And that's a really important part that has led to a lot better analysis in the activity world.

If you come back to Lawrence Korsmodel around who does what with beam and it's the same thing, we go, who does what? So we'd see employee launches class, there's no customer in there. There's no customer in that who does what. So then it might be system notifies customer, that course has been launched. Something. Something horrible like that. Okay, so we bind the activity schema table to a core concept. So if I use again, my customer orders product and we look at an organization that actually manufactures the product before they send it out, we're going to see another activity schema table which is bound to the concept of product. Because we're going to see maybe products being manufactured, products put in warehouse, products shipped out to customer. So we're going to see those activities because we of the fly, we're watching that product and we're watching what happens to fulfill the order, all the activities or steps that happen to that product before that process is finished. Did I get that right? So we end up with a bunch of core activity schemas. And what we know is actually if we look at an organization that's been modeled well over time, there is probably somewhere between 7 and 20 core concepts, core concepts in that organization and a really complex organization.

And.

And we also know that actually customer is always the primary concept. If you're an organization that deals with people, you might call it customer, stakeholder, citizen, whatever, But a person or an organization, and we get some complexities, there is always the driver of most of the events with the activities. Is that right?

Yep, exactly. So in narrator world, you would have a product stream and a customer stream. They might have some similar activities. Like a customer might have their product manufactured and a product stream might have started manufacturing. And the reason why that's really important is when you're asking questions about your operational excellence, you might say, how long does it take for a product to be manufactured? Or how many times is a product moved in a warehouse? And based on those questions, you'll go to the. Based on the entity you're trying to change, you will go to that stream. So if you're trying to change the manufacturing processes, you would look at the product stream. If you're trying to impact the customer, you would go to the customer stream. It's okay for activities to be in both streams because a customer's product also might get started manufacturing a product might complete manufacturing. So many things that happen.

Okay, so someone there that I missed. We all know that often we get given a question where there's no data for it. Why are highest value customers buying that product at 2am in the morning? We just, we don't have data for that. We don't know. Good question. Let's go get some data. So that, that example that you used that product so customer has product manufactured for them because we're binding this one to the customer activities, often that's not available. That's not data we store. We don't actually say when we're manufacturing a product unless it's a product on demand for that customer. We typically manufacture a bunch of products. We put them in a warehouse and then we pick them and send them. Amazon doesn't say this book was manufactured for Shane Gibson. It's in the warehouse and they're picking for it. But if we can actually say that product was manufactured for a customer, that's the way our core business processes work. Then we inject it both into the product activity schema and into the customer activity schema because why not? There's no cost. And now the ability to query that is much easier because we don't have to cross join across those two activity schemas. Which is where I got to was my next question is actually joining those activity schemas must be a nightmare because whenever we do any joins of any data that's at a different grade. Okay, so you don't, you don't and.

You never need to because you shouldn't ever. So like you see this a lot. Let's go through some examples. Can you come up an example where you might think you might want to join two streams?

Let's do the standard one which is I have a prospect coming in to the website and I don't know who they are, but I have some anonymous prospect ID because it's Google. Then they come in and then they sign up for a form and so we convert them to somebody we know via an email address and then we'll go out and we'll ping them with a marketing campaign. So now I combine that to my customer and then they come in and buy something and then we ship it and then we do fraud detection on Them.

So far, everything you said is still related to a person. And in narrator, there is a concept of a customer and what's called anonymous customer id, which is if you don't know, you have your global identifier, which let's say it's an email. And they might be a credit card number, they might be a name, last name, zip code, they might be a cookie, they might be a prospect id. So it's still the same person. But taking your example one step further, let's say then they sign up as a company and now they're a company or an account. And an account pays an invoice. And you want to know from that person who first signed up, do they pay? How long take them for that first person to pay. In that situation, which we do a lot, you might have the activity, the person's Ahmed who signed up Ahmed might have Ahmed. Company pays bill. So you might have an activity called company pays bill. The same goes with the company, the account stream, or the company stream. We might have employee received email or employee submitted lead. Or we can have those concepts from the perspective of the entity you're trying to change. Which makes it super easy to query. Because when you're asking questions again, your questions are often, I want to know like, does that person who comes in, how long till their company makes a payment? Well, great. I have company makes payment, submits a lead, and it's in the same stream. Super easy. Everything is nice and clean. And yes, when I'm building those activities, I might in my backend database have company and company has users. And I might join on the company user and I'll just add an activity for every person in that company at that time that they made a payment. It's okay, because from the perspective of that person's individual stream, it makes sense. And that's why we build this thing on top of a warehouse. You mentioned activity versus event. I use the unique words so people don't get mistaken for the way that you naturally treat CDPs where you have to log the events. No, during a database, you can combined, you can create that company pace. You can put that on every single person streaming and on every person who cares. You can do that. And it makes it really easy to answer those questions.

Okay, so that's key though, is that where we see an event come through. And using other modeling techniques, we'd ask who does what? Where we say in that event and that core business process, what concepts are involved? So customer orders, product. We have a concept of customer concept of Order concept of product. We bind that together. Right. Because it makes sense. Because we're not going to ask that question, what an activity schema. What we do is we may end up with a customer activity schema, an order activity schema, and a product activity schema. And we would push that one activity to all three of those schemas because they're bound to it. And now we can ask the question between. Okay, I get reading it. I haven't picked that up.

Yeah. And just to make sure, like, it's really rare that you actually have an order activity stream or a product stream. It's only when we work with Costco, like company, they have the manufacturing process where they're asking questions about the manufacturing and they're asking questions about the person. For a typical E comm, it's usually just that person's dream because that's all you really care about. Everything you're asking is from the perspective of a person and you want to know, like, give me the outliers of when people take the longest to get their product delivered. It's all relative to the person, actually. You don't give a shit about the products. You care about the person who's waiting for a long time. Yeah.

But what we know is when we look at organizations, we focus on administration events. We focus on accounts clerk entered invoice, accounts manager approved invoice, somebody released invoice. We focus on those sub processes because that's what we see people doing. And if I. And again, let me just play this one back to you. So I'll use a different example. Insurance company.

A customer applies for a cover. And then we know there's an actuarial process actually going through. But effectively, when we think about it as a core business process, we always bind it to the policy. Customer applies for cover, we create some form of policy. We then get the actuarial team to go and look and say, are we going to approve that policy or not? And what we don't tend to do is we don't tend to end that off with a customer name. We tend to divorce them from that activity, even though it relates it. So all we're saying is actually, customer has policy reviewed by actuarial team.

Customer policy is updated.

Yeah, yeah. And we know that policy relates to them.

Yeah. And then you can answer really complex questions, really easy. So, for example, as a policy gets updated, you might want to know, every time the customer is meeting with the sales team, the customer starts an opportunity. A customer has a meeting. Customer might go into discovery A customer might have another meeting and I want to know when they had that meeting, what was the policy revenue, for example, this is really easy to do in the activity schema because you say when a customer had that meeting, what was their last before updated policy? And I can then see the revenue there. So even though the policy might be going through a separate experience and a customer is going through a separate sales experience because they have customer start opportunity, customer entered discovery, customer had a meeting, customer had their policy updated, I can always say give me the last before policy updated. And now I have this nice table that makes that really complicated question really easy.

And then again, we'd probably say that in the type of industry you're in would drive what your core concept is. So we're doing some work at the moment in the utility market and it's not about customer at all. It's about asset, physical device, and so that's that. I can infer there that actually the majority of their activities will be bound to that one schema. I might have some secondary schemas, but actually everything we talk about is around these assets, this bit of equipment. We don't talk about customers at all because it's not for that part of the business. It's irrelevant.

We had buildings and different things. My building gets acquired or building goes for review or building gets opened and all sorts of stuff.

As we're building out more and more customers on our platform, we're actually looking at the metadata because we're really intrigued by this question. Based on our consulting experience, we have this gut feel that from a who does what, from a concept point of view, that most organizations, when well modeled, have somewhere between three to seven, maybe a maximum of 20 core concepts, if they're a really complex business with actual multiple businesses embedded within them, multiple unique businesses under the same umbrella. So your point of view, how many activity schemas does an organization typically they have a core one, which is customer, and then what they have normally a 3, 5, 20. What's the rule of thumb for how many others you've got?

I think Most companies have one. I think probably 20 to 30% have two. I think the maximum I've ever seen in all the customers has been four.

Okay.

It's really rare that you have more than four unless you have multiple product lines. I would say every product line usually has something they're trying to change. And it's really rare that you have so many things you want to change. Even WeWork, which had many product lines, there's people. There was Companies, there was people, company building. That was most of things. And then I have internal stream which is like employee gets hired employee, that's a computer. An employee gets for all the people, analytics. Do you have other things? Maybe it's just less common. And the point of it is not to really try to think about it. Like the whole point of activity schema is you don't have to decide those things now. You can always add another activity. The whole point of it on warehouse, you can always. One activity can never be used and you can just delete it. You can always add another stream. I would like to always say is start with the questions you're asking, see what the core entity is. You'll just keep going. And then if you keep feeling like you're constantly trying to add new data, add an activity, it's fine. And if you create an activity that is never used, just remove it and it's fine. And because everything is on top of the warehouse, we're just really remodeling the data in the warehouse. You can always add and remove and change your mind. Your data is not lost. It's just captured in the warehouse. So you can just kind of go in and ask and answer questions.

Okay, so let's just replay the bic. What we see with any modeling technique is there's always a bunch of anti patterns. So within data vault we go multi active sat. Where in the satellite there's more than one active record that is an anti pattern that causes you a shit ton of problems. Star schemas. If you have multigrain facts, you're dead. So don't do it. So what you're saying is if you're using activity schema and Your organization has 20 activity streams, maybe have a look at what you're doing. Because you're probably taking an old modeling technique and just trying to replace it with activity schema.

Exactly.

The second pattern that in there is. This is not one data modeling pattern to rule them all. Right? Because what you're talking about is the data is in a warehouse and now you're applying activity schema to that data. So you're not saying that you get your data out of your system, the capture, and it goes straight into an activity schema, and that's the only data modeling technique you use, is that right? You're saying that you may use other modeling techniques if they have value, but the activity schema is the core data modeling technique you use to ask any question, is that right?

Exactly. So we like to be in the warehouse. You can Add dimensional joins on the activity schema, of course, if that makes sense for you. It's just exactly that. It's just the core and it's designed to ask and answer questions. You talked about anti patterns. We say if you build more than two streams, you should ask yourself why. If you have more than 30 activities, you should stop and say why. Most companies don't have that many. And then three, if you find yourself taking more than 10 minutes to answer a question, you just stop and say why. And if you're spending more than 15 minutes to define an activity, you should stop and ask why. Literally these are alerts that we have in our product where if the user is opening the transformation editor and they're spending more than 15 minutes to define an activity, we will put an alert. Usually this query should be simple. It should be really slight mapping, not really complicated things. And then everything else happens in the activity schema that makes it really easy so you don't have to worry about it.

That's good because again, when I looked at it, I thought for me, Data vault is the middle combining layer where we combine data from multiple source systems into one place around those concepts. Made sense to me within activity on schema on top, where I'm effectively making it incredibly easy to query that beast of a model. But I'm still going to think about it a bit more because I'm not sure why we need the complexity in the middle. I need to go through that.

Most our customers don't do that. Most customers go production systems, build activities on top of them and then go and answer and it becomes really nifty because each activity is often one concept. If an activity has more than one concept, then you're like, well, that should be two separate activities. Stop doing that. So it makes it really. It shouldn't be like a trivial mapping that you put to that. You're just restructuring your data into the model.

Okay, so again, if we think about core concepts and who does what in those business processes, like I said, there's always three to seven concepts involved in a relationship when we watch it happen. And again, Andy pattern, you get to 20 things in that relationship and then you've got your modeling wrong. So if we think about that, one of the things that I struggled with was yes, I hold the core concept of customer or person against that activity stream, but I naturally want to add the other concepts in. I want to do group buys, I want to report by those things. So I naturally want to bind it to there was a product and there was an order against that activity event. And the activity schema was limited to I think three. And then I remember you came out with somebody, I think it was you came out with something that said you've gone into some larger customers and you know that didn't scale. Because we know that once we get these modeling techniques, we then get all the edge cases where it's okay, nine times out of ten it works. But if you happen to be doing that one thing as a company, then we do it. And so talk me through that. These secondary concepts that are bound to that activity.

Yeah. So the way we designed it, we said, okay, most activities have three features, so we'll just maintain three features. And then we said if you wanted to more than three features, you would just do a normal dimensional join to add the additional features. Just traditional. So those three features really, this was designed seven years ago. Back then JSON and unstructured data wasn't really common. People didn't put that in their warehouse. So, okay, three features. For most things it's really fast to query. And then a join when you need to do additional is fine. The honest answer is people got really obsessed with the three features. And then what if I want more? And I'm you don't really need more. And you have dimensional tables. Like I can tell you how many companies out of the activities it was. 4% of our activities had dimensional joins. Most people didn't need more than three. So I was okay, you still have your dimensional joins to use and we support it, but you don't really need it. But people were really upset. So we're like, great, let's just make it a JSON blob so you can add as many features as you want. Part of that is you talked about anti patterns. We do a lot of checking for this and recommendations and our whole system is designed to help them avoid this. But in order for people, I want to know the at source of the order and I want to know the attribution of the order. You're like, no, those are derived. Those are shoving things in features. An order happens. What is related to the order only. But the ad source where we acquired the customer. Great first ad, first ever started session. Grab the ad source. Don't shove that into features. And people have a tendency to create hundreds of features and you don't need that. You only need the features that are unique to the order because you can always borrow features from other activities. So if you want to know how many people often will do, they'll have, I don't know. There'll be a lot of things that you might think you need, but you don't. You want to know when a customer purchased the product. They want to know how many inventory we had in that product. Or you might, that might make more sense to put it as when the customer views the product. So you might say, give me the last view. So we allow you to now have a JSON blob where you can add as many features as you want. We try to recommend that you don't put more than three or five. We still support dimensional joins, so you can still do that, which is helpful if you have a table that you just want to link to, but we try to avoid. We still give you the flexibility to add as many as you want with the JSON now and unstructured data, but we try to recommend that you don't shove like a million features.

All right, so let's unpack that because there was two patents in there. So the first pattern was, I've got somebody coming through to the website doing a page view, and I want to attribute where they came from. And our natural reaction is to store the attribution of which ad they came from against that event. But what you're saying is it's an event in itself. Customer viewed ad and then customer came through and viewed webpage. And now we know they viewed the ad at this time and they viewed the web page at this time. And therefore those events activities are correlated. So we again, we denormalize the attribute back to being a core activity itself. And then, so that's the first pattern. And again, if you've been trained in any other type of modeling technique, you, your brain works as they're just attributes of something. They're not core things. So you've got to untrain yourself and go, the attribute's actually an event, it's an activity. Okay. And then the second thing is, which was my next question is in the way we talk about core concepts, customer orders and products, and we talk about detail. The customer has a name, the order has a value, the product has a sku. And so that detail is important. So what you're saying is in the original pattern you would have three features, and to get the detail of the customer, you would join out and grab that detail from another table that describes the customer name to make it more flexible. Because we hate constraints. Whenever we see a constraint, we think we're losing something and we focus on, oh, it can't do that. What we do now is we effectively land those Concepts as a JSON object. And so the pattern technically is pass the JSON object, get that core concept, and then bounce out as a join to the detail table to say, okay, that thing has a whole lot of attributes or detail about it that relates to that concept within that activity. So, yes, we can answer that question out of that activity schema really easily. But there's still a little bit of complexity, because every modeling technique has complexity. And so this one is when you want the details, you've then got to go and go out and get the details from somewhere. Okay, that makes sense.

Yeah. So, yeah, you have to open the JSON blob and get the details from the JSON blob. So the activity schema is. It's 90% of the stuff you go to. But there's also what we call our customer dimension, which just has the name and the age and all the stuff that you put on the customer. You might have aggregate dimension, which is like, if you're spending money, like I spend $30,000 on this campaign on this day, and those can be joined on top of the activity schema when you need it. But, like, most of the stuff you're doing is within the activity schema and you just unpack that JSON and pull whatever features you need from it.

Cool. So one of the things I love about Data Vault is because there's so few objects, there's so little code for every. For 90% of the cases. So we talk about Create Hub, Create Set, Create Link, where we create those tables, load hub loads at load, there's only really six bits of code. And the benefit of that is we harden that code, we use that code every day. That code gets immutable. Eventually we stop playing with it. And so what I can feel about activity schema is that's the same when we breach out to those detail tables, those what you call dimensions, is that we know it's past the JSON field. Find that feature, look that feature table up. Because the naming should be the same. If I make the name of that table the same as the name of the feature, value the feature, now I can automate it.

It's embedded in JSON. So it's just like unpack JSON. We actually just literally store a JSON blob called feature JSON on that table. So in that row, you just unpack it. Yeah.

So if my features are named as the same as the dimensional table that holds the detail, I just now have one bit of code which is unpack that JSON, go look up that table get the detail. If I was writing it myself.

Oh, the details would be part of the JSON so you don't have to look it up. All the details are in the JSON blob. So the customer dimension, when you join on customer, that's like standard domain. And then on the activities, like the order details would just be in the feature JSON on that row.

So order value would be in the JSON.

Yeah. Okay, so you don't have to look at it, you just literally unpack it. It's right there.

Okay. But you're doing it as a nested JSON object, so you're effectively embedding detailed tables for each of those features in that JSON object.

Single dimensional. You never make that JSON object have an array. So within that order. So if you're used to snowflake, you would just say word Activities Order Feature JSON.order value and you get the order value feature JSON like tax amount and that gives you a tax amount. And it's like that. So it's really easy to use. We don't let you devil nest that. It just makes it unnecessary.

So if I play it back effectively, you're taking the dimensional attributes and you're binding them as a JSON string against the that activity in the activity table. So I'm not going out to another table. And then how do we deal with change then? So customer changes their name.

Yeah, great questions. And now we're like in the realm of the paradigm. And then how you work around some of these things, there's two ways of doing it that we support. One thing is that if someone is. If it matters to track change or it doesn't matter. So let's say you. So remember you have a customer has an attribute table and you might just update. That table is often materialized, viewed, and that table might just have the updated name. But let's say if customer changes something that's important for us to track. So you might have that as an activity. So customer updated their location. In the narrator world, which is activity schema with the edge cases. We do support what's called a slowly changing dimension where you can say, okay, I have this other table that's customer credit card and when updated at and narrator can take that and then do in the background for you a last before join and bring that into and make it look like a dimension. So there is those additional cases. But yeah, in the pure activity schema way you would just do like give me my customer activity updated location.

As I said with every data modeling technique, there's the core things it does really well and then there's the edge cases that always bite us in the bum. And the key is understanding those and dealing with them. Okay, look, this has been awesome. We're pretty much running our time, so I've got a couple of quick fire questions. So I haven't done this before, so this is going to be fun. There must be a standard type or a number of events that you see on a regular basis for specific use cases, specific industries.

Yeah.

Cool. Do you publish those openly?

Yeah. So we're beginning to do that with now for Ecom and we have that as what we call packages. And in next year we're going to actually open source. You'll see it in our Doc site Docs narrow AI. We give you the common activities per industry. We also have realized that people just like starters. So if you're an E Comm company, you come into NARRATOR and you have Shopify, it'll spin up seven activities for you and you can go in and edit them. We also do that for spinning up dashboards and analyses that are just common that you do. So idea of NARRATOR is you get started and we would spin off those activities, those dashboards and analyses and we call that a package. So that's an E Comm package. And starting next year, NARRATOR is going to actually allow all our current customers to upload their own packages or on analyses or on dashboards. So in a world where everything's on top of a standard structure, people can actually begin sharing their work. Because even though you might be your 2e comm companies, you still have completed, order, ship, product. We can talk about it and we can do analyses on it. But then when we go to implement those analyses, we often have to deal with everyone's unique data. But in the activity schema world, everything is the same. So even though you have unique definitions for what an order is, everything else for the analysis is the same. So we're actually allowing our customers to begin sharing those things and, and buying and selling them across companies and just make that really easy. Excellent.

And then we all know that we start off with one data engineer if we're a small company. And the standard joke I have is I've never seen a one person data team. So we always go to three to five, create a team and then we see some success and some value, but we get this massive backlog. So we then scale and we go to three teams, right? And then we have to figure out how to break the teams up as we're going to do it. Pipelining, where it's an awful, wonderful process of requirements handed over to build, handed over to QA or are we going to domain bound them? So we go, you're the finance team and you're the marketing team. So activity schema, if that's the core modeling technique we're using, that's going to drive a lot of our scaling patterns.

Yeah, it's like you said, we don't see one or two person data teams. I see a lot of one or two person data teams. I actually have seen many companies. Our biggest leads come from companies who just laid off half their data team. That's where usually they get interested in the activity schema or narrator. And usually because again, your definitions are so simple and most of the time you're just reassembling those definitions. We often see answering question speed is 10 minutes. A single data analyst or a team of two can handle 100 questions a month, which is unheard of. It's just by having a simpler data modeling paradigm, so many things go away. And so we see so many smaller teams doing a lot more work. A lot of work. When Narrator had a consultancy arm, we used to have one data analyst per 10 companies. That was the ratio that we were doing. And you can see that value in just that repetitiveness and the reusability of the activity schema.

If we were going to scale the teams or we're going to distribute them out into the business because we wanted to go to more decentralized distributed model, what we'd do is we'd do it around the activity, the core concept of the activity. So we'd have a team that looked after customer activities and we might have a team that look after manufacturing activities. I mean, the key handoffs would be when the manufacturing activities were defined and they had customer in the sentence, then they'd notify the customer activity team, hey, here's another event that you need to do. Can we push it for you or do you want to go push it yourself? So that's the shared languages where your activity actually has one of the cool concepts for another activity team. You just let them know that you've got this thing that's useful for them. Okay.

Yeah. Also it's very common for you to have, because it's a SQL and these are actual tables in your warehouse, you can just literally write a SQL query that wraps around the other stream and brings the data in. That's really normal too.

And so one of the ones that I'm always Intrigued on is this pattern of data mesh which I was really excited about in the beginning because I thought it was actually going to change the way we work. For me, everybody's just buzzwashing it now. But if we were going to mesh this, if we were going to push this back to the software engineering team to do the data work without us, this would be helpful, wouldn't it? Because effectively, once you taught that software engineering team what the activity schema is, they're just going to actually populate the activity schema table out of the software that they do because they know they're building e commerce website that relates to customer and they know what the events are. So we should be able to teach them to use this technique and have them do the work without us, shouldn't we?

The answer is yes and no. In theory that sounds great. In practice those people are really bad at defining activities. So the reason why we shifted away from people like streaming events in we always get that. Why didn't we just build a CDP with all this extra power? It's a super powerful cdp. And the reason is that software engineers have bad practice of track everything. They just want to log openmodo and you end up creating so many garbage activities that are so nuanced that no one uses them. So I actually think about it backwards which is if you have a single data analyst answering your questions as you get questions, start defining activities only as questions arise and then you'll see what activities you actually need. Because so much of that data can exist in your database already most of people are asking about primary database things so you already have that data and you can build activities and they end up reflecting the concept. An example is wework didn't have in our entire database systems the concept of an office or a desk and that's what we sell. But wework in the engineering team thought about it as reservables and reservations. And the whole point is to make activities as closely resemble the way that customers are asking questions, not how they're represented. So we are very much for don't start from what you built and say let me model it, start from what people are asking and build models that represents that language. And therefore, because you're going from the stakeholder's perspective, often I don't know who you called on your shoulder observing everything. That observing person doesn't give a shit about what's in your database and what your engineering is calling it. So I would lean toward avoiding that. If you have an org where the software engineers are super tied to the business and they're really understanding, then it makes sense. We do have CTOs of small companies that use NARRATOR every day. The flexibility of building business objects so they don't have to query, building more stakeholder representative activities and then answering questions that way.

And last one, ChatGPT. Because we can't do any podcast now about ChatGPT, so we're seeing that everybody's Runway's gone. Unless you're a generative AI product. We see every incumbents and every new startup doing text to SQL.

Yeah.

And so one of the interesting things for me is whenever you watch those demos, you sit there and you go, that is some lovely curated data.

Yeah, exactly, exactly.

And we, and we know that joins are difficult and it's going to be interesting to see how any of the generative AI products deal with that. But again, one of the benefits of the Activity schema, I could guess, is we've got one big table. So in theory, ChatGPT will love it.

We do. And there's three currently open source projects that use ChatGPT on top of the Activity Schema that make that really nicely. Narrator's building our own version, which allows you to do a lot more complex questions. Bart Moses, founder and CEO of Monte Carlo, recently wrote an article about the only way to get generative AI to ever become very popular in data is to have it on top of a standard schema. And she references NARRATOR and Activity schema because that's currently the only real standard. Everyone else has modeling paradigms and they let you build whatever tables you want. You have to think about it, and it's up to the engineer or the data engineer who builds it to represent it. Activity schema is like, no, no, no, no. There's rules and there's one way of structuring the data and you're going to follow it. And because of that rigid structure and the fact that you're only using activities and temporal joins, it has opened the world for a lot of tools like NARRATOR to exist, but also chatgpt to be able to. There's just such a controlled space. And the language that the human is asking questions in is modeling the same language that the activity schema uses. So you're using words like in between and first and last and made payment and when the customer called us. And so it makes the activity schema is I believe to be the only way that ChatGPT will ever work on top of data. And I think a lot of people will share that belief, is that this is a standard structure. And the more people that abide by that standard structure, the more we can share, the more we can reuse, the more we can have ChatGPT on top of it. And I believe in, that's what I believe in the world. That's why I started it. My first ever raising was about sharing and reusing your analytics in your work. And it has to be on top of a clean standard structure that still supports every company to customize their definitions. And I think Activity Schema does that really nicely. It allows you to have full control over definition, but still have concepts that everyone can use and reuse.

I think I agree with you partially and then I'm going to disagree with you partially. I agree that the more you get highly opinionated products, the faster and more reusable they become. And then you always got to watch out for the downside, which is by being highly opinionated, you're removing some choice that has some consequences. But that's okay because you get the benefit of it's faster and it's more bulletproof and all that kind of stuff. And so for me, with CHAT GPT, your highly opinionated structure is the schema itself. I think there's another pattern which we'll see a lot of, which is that there's config or metadata which holds the opinion and the data structures themselves are a little bit more flexible. But then the same outcome because we now have an opinionated later that that holds the context, the semantics, and then hints all that to ChatGPT, so it doesn't have to worry. So I think these two implementation approaches of the same pattern, which is standardize things and therefore the engine knows what to ask in a repeatable way every time.

Yeah, 100%. And I think semantic layers, I'm like a very big fan of. And the thing that I would always say and add in this context is that if you're a company that's answering a series of questions that you can define a semantic layer on top of, you should do that. You should definitely do that. I think that the things that we notice with especially fast growing startups is that the questions are always unique, they're just off the top of the head. So sitting and trying to imagine the semantic layer ends up having way too. Like we talk about Airbnb's like data dictionary and their metrics layer, they have 10,000. At that point you can't differentiate column 972 and column 322 because the words seem so similar. So at that point, semantic layer is garbage. The activity Schema is really good for open ended, fuzzy, repetitive questions. Once you build activity schema and you start answering questions and you're doing it, some of those questions you just want to fucking put in a table and just reuse. We allow you to do that in narrator, materialize it, put it. The whole point is when you have a new question or you want to add a dimension or you want to slice by this and you want narrator makes that really easy. And activity schema lends itself really nicely to that chaos building with more and more questions. So if you have a clean structure that you want to represent and model, go ahead. I think that's what some of these paradigms help you reuse. If you have a lot of new questions and there's fuzziness and there's dynamicism and you're like dealing with a chaos situation, do it. So yes, if you're Nike and you have quarterly metrics that you're doing or you have CFO reports that you need to be very specific, don't use activistima for that. Those are CFO reports, they need to be do it. But if you're a machine learning team or you're a data analyst or you're at a startup where your company is growing really fast and you have hundreds of data requests, that's when the activities team really lends well because that speed is really important and you don't need that, you need that. Flexibility and speed just will carry you through victory versus the world of a CFO report where like formatting and organizing and it's very specific that build it, use a SQL query, put it in a semantic layer, model it like do it normal way. It depends on which modeling technique you want to use. But yeah, for fast questions, iterating, answering questions really fuzzily, there's nothing better than activity schema.

It comes back to that there are patterns that are brilliant in a certain context and there are patterns that are harder in other contexts. And these patterns where actually it's an anti pattern. If that's your context, most people it doesn't work for. I still say have a go because you might be lucky. But the idea is experiment. And so again, one of the things you said a little bit earlier and I didn't reinforce it, is this idea that it's an agile modeling technique. You start small, start with typically customer activity schema and then add a few events, few activities and then add a few more. And then if you need to break out to a second stream, do it. And then if it then doesn't have any value. Kill it. Yeah, cat or not pets. That's what we're after in data. Excellent. Hey look, that's been great. If people want to get hold of you, how do they find you and talk to you, see what you're writing about, find out what you're doing? What's the best way for people to get in touch?

Yeah, you can add me on LinkedIn. My email is just ahmedarrator AI. Feel free to ask email me any question if you come up with a edge case that you're how does Narrator work in that case? Shoot it. I love those edge cases. And three check out narrator narratordata.com where you can experience narrator sign up. Try it.

It's designed to be the activity schema plus all the edge cases that we found from working with hundreds of companies. So I'm excited to interact with every single person who sees this.

Excellent. Thanks for coming on the show and I hope everybody has a simply magical day.

Awesome. Thank you.

Data Magicians was another agile data podcast.

If you'd like to learn more on applying an agile way of working to your data and analytics analytics, head over to agiledata IO.