The Activity Schema data modeling pattern with Ahmed Elsamadisi - AgileData #42

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Ahmed El Samadisi.

Hey, Ahmed, thanks for coming on the show. Today we're going to go into another one around data modeling, this time Activity Schema. And I'm mega excited because I've read a bit about Activity Schema. Had a little bit of a play with it, but nowhere been in the weeds. I'm looking forward to understanding what the pattern is, when we should use it, more importantly, when we shouldn't, and diving into it. But before we do that, why don't we get a bit of a background from the audience about you, how you got into this lovely world of data.

Awesome. So I actually started my career in self-driving cars in 2010 before it was cool. And so got to spend a lot of time focusing on AI and decision-making. Moved my way into AI for missile defense for the US government, and then eventually joined the startup WeWork and built out their data team and watched it scale. So I got to see data used in all sorts of different ways, from mission-critical applications to business decisions. And what I found was that having implemented the system at WeWork, we went through maybe 4 or 5 data stack implementations. We kept running into issues. And couldn't find any good solution. So what we realized was that everything was always coming down to data modeling. And data modeling always is the problem for why data scales terribly. And I think a lot of people nowadays are beginning to know that with dbt publishing an article about how their data modeling didn't scale and so many things of that nature. So I'm excited to show this paradigm that we've been using, a lot of our customers use, and, It scales really nicely and hopefully more of the world can experience it.

We know that bringing back data modeling is kind of one of the themes this year and make data modeling cool again. For me, you've come from working on self-driving cars and working on rockets to data modeling. How cool is that, right? You've gone from being a rocket scientist to a data modeler. That's a progression. Before we jump into Activity Schema though, let's just, 'cause we tend to go all over the place on the podcast and that's all pretty cool. We always come back to the core thing. But that idea that at WeWork you went through 5 data stack changes. Talk me through that. Was that because there was new, cooler technology out? Was it because the first 4 didn't work? Was it because you were just iterating? Because that's a fairly expensive change, depending on what you mean by changing your data stack.

Yeah, so I feel like a lot of people actually go through the same flows. We had a Postgres database, Tron Scheduler that would schedule our queries and materialize them. And we had Chartio for visualization. and we're like, this seems great. And then it, we just had so many things scheduled, answering questions is really hard. And the company was like 200 people then. So it wasn't really that much, that many questions. And we're like, aha, I'm going to upgrade my database. I'm going to switch from Postgres to Redshift, from Unicron to, at that time, Luigi was popular. So we used Luigi and then we had Chartio and still more and more data models still can't answer questions. And we're like, ah, I know what it is. It's not Chartio. We need Tableau. So we got Chartio and Tableau. And we're like, that's gonna enable us to answer more questions. It didn't. So we're like, okay, probably this instead of Luigi, let's switch to Airflow. And so we had Redshift, Airflow, and instead of Chartio, we switched to Looker. So we had Looker and Tableau and we're like, that was gonna help. And then we're like, people are still yelling at us and numbers are not matching. So we added Anodot and we're like, that's gonna solve the problem. We added a dictionary, we added some lineage, still numbers don't like, Numbers still don't add up sometimes. People have so many questions, just like the backlog. We're like, aha, it must be Snowflake. So let's switch from Redshift to Snowflake. And then we're like, fuck Airflow, let's build our own custom scheduler that does our own things. So we built something called WeModule and Data Manager. Oh my God. And then we had Looker, Tableau, Chartio, and then we had another tool, somebody bought Wave Analytics and we had, it's just so many tools. Like it was just so many tools. And we had a 45-person team and we still had the same problems. People would have questions we couldn't answer. Oh, we also bought Heap and Amplitude to help. And still people had so many questions they couldn't answer. Everyone was complaining. A lot of the dashboards didn't add up and it was just a shit show. So we're like, why is that the case? So we actually went on a tour and this is the interesting thing is I went on a tour, I talked to a bunch of big companies like Airbnbs and Netflix and Spotifys. And I was like, how do you guys deal with this? And it turns out every company is in the same boat. Everyone is just refactoring the same things. And every time you refactor, it seems great for the first couple months. You're like, oh my God, I just threw away— and then you refactor and you, or you buy a new tool and you just move all the stuff that's already working. So every tool appears incredible for the first couple months, cuz it doesn't have all the shit yet. So that was exciting. So all these companies were just like, yeah, that's the job. The job of data engineering is to manage the shit show. That's it. There's no solution. You just try to minimize it. And all these tools are helping you manage this growing chaos problem. And what I wanted to do is, is there a system that's designed for why does chaos of data continue to grow, understanding what causes it, and then understanding What part of this whole data stack is the real problem? Because as you dive into it, you realize that Snowflake and Redshift and BigQuery, like the slight differences don't really matter. People love talking about it as a millisecond faster. I don't care. I have these things being materialized and then people talk about the query editor and whether it's synced to GitHub, embeds Jinja. And it's like, okay, but yeah, but it doesn't really matter. I still am writing, compiling down to SQL. So, okay. So I've used all these different tools that haven't really changed our approach. I use dashboards and whether it's purple or Chartio or Tableau, whatever color their color scheme is, doesn't only make the solution better. So we're just constantly using new tools and using those tools as an excuse to refactor. But this fundamental approach doesn't work for at least what I've seen. I just couldn't find a single person who was like, "Yep, a year into my stack and it's great." Everyone's like a year in and they're like, "I wanna refactor and blow this thing up because it sucks." So that's where my whole life was like, okay, I need to really dive into this problem and let's go figure it out.

Yeah, it's really interesting, isn't it? It's interesting as data people and as technologists, we love to refactor our stack. We love to refactor the tools. We love to invent stuff, not invented here, but we hardly ever refactor our way of working. We sit back and we watch our data flows. We watch the data move and we go, oh, there's a bottleneck over there, we're gonna go and We're going to go and refactor that bottleneck. But we hardly ever sit back and watch the way our team works. We don't treat humans as part of the system and go, where's the bottleneck? Where's the bleed? I was lucky enough to work with a guy called Sean Maguire many years ago in my consulting company, and he now works for Dataiku. And he was running, before that, he was running an analytics team over in the UK. And he told me the story, he went to a conference in the UK, meetup, standup, whatever. On the stage was one of the FAANG companies. Can't remember which one again, Airbnb, Amazon, one of those big companies. And he said to him, he said, how do you deal with the data problem? And the person on the stage said, what do you mean? And he goes, when you want to find some data, you've got so much of it, how do you find it? And the data scientist said, well, it's easy. I just go into Slack, I log a call, 15 minutes later, the data team have got me the data. And so they had hundreds of people that served the data to the person that needed it, and they scaled with bodies, not with ways of working and not with systems. So for me, that's fine if that's your business model and you can afford it, but for Most of us, we can't, and it's not overly efficient. And if you think about that 5 changes stacks, at what stage did you bring in data modeling?

First of all, the story you had is such an incredible highlighting of the problem, which is even the greatest, the most advanced tech companies that sell you these software to do all these things, they just shove people at the problem. It's their, whatever, I'll just have people go deal with it. So that's just kind of an interesting insight thing. But the question I think is always going to be, why? Why is it hard to find data? I have 1,000 tables. So which table do I use? Why do you have 1,000 tables? The idea of the activity schema, which I'll bring it to that, is what if you had less? So like if you had one table only, you would say, where is the data in that one table? Problem solved. Cool. If your data was standardized in the structure, if you were like, what data is in the table? The structure is consistent, so you don't have to ever, what does this column mean? Once you have one table, then data lineage goes away. The lineage, one. Data dictionary goes away because the table should be self-explanatory. It's consistent. Instead of going and messaging someone to be like, where do I find data? It's, I only have one place to look. So having the concept of one table is really kind of nice because it makes things so much simpler for processing, for for finding, for the way that humans actually deal with data. Now the question is, why doesn't everyone just build one table? That seems like a simple solution. And I think to understand that, we have to go back to why do we build multiple tables and why do we build them? More interestingly, why do we build it the way that we currently do? And that was like, my obsession. So the reason why you build multiple tables is you have different questions. So you're like, okay, cool, every question has— you have to answer the questions and therefore you need data. But you might think, why don't I have one table for every concept that I have? Because the company doesn't have many concepts. And then all I have to do is combine those tables and I'm problem solved. And I think there lies the problem. Combining those concepts into tables that are used to answer questions, that's what like 95% of your tables are, the combining. And with questions and the way people ask questions, there's constantly new questions that relates data differently. And because you're constantly asking questions that are unique, you're constantly relating data in a unique way, which means you constantly have to build a new table to represent that data. So your last, the follow-up question is, okay, then doesn't SQL allow me to do that? And then, so this is where the data splits off. So the star schema was designed in a world where you had one data system. When it was designed, you had every company had one Postgres data. Storage was very expensive. Compute was also expensive. You would take your data from one database and build aggregate tables so you can actually visualize X and Y. Because you're working within one system, you had foreign keys. If anyone who's listening to this, a foreign key is just a way to join data with itself. And the questions people were asking was, how many comments are on my ticket? And therefore you join comments and tickets, aggregated, and it makes perfect sense. Today's age, every single tool has embedded analytics. No one's asking your data team how many comments are on a ticket, like Zendesk, Or the ticket tool will give you that. They're asking more behavioral questions, which is, okay, when people submit a ticket, are they more likely to churn? Or what part of the app resulted in people submitting the most tickets? Shit. Now I have two separate systems and those systems don't talk to each other. So you need to write some sort of complex query to combine the data in a way that answers that question. This is the separation between writing what people talk about, I know SQL. It's like, sure, and every single BI tool can generate SQL. But the SQL to combine those two sources is very complicated, is very nuanced, it's never predefined. That's what your data team is spending all the time doing, is combining that. Because it's so hard to combine it and requires so much SQL and code, you end up saving that table and that's why you end up going to a person who knows exactly what makes that table, and then they can tell you, here's a table to use. That's where we are today. Does that make any kind of sense at a high level?

Yeah, look, there's so many patterns in there that I just wanna unpack. So where do I start? Where do I start? Okay, so let's go to the star schema one, because really it intrigues me that the US is the king of star schemas, and actually we see Europe as being more data vault modeling. And what's intrigued me is when dbt started out and they did that, awful thing because I'm hyper-focused on terminology. Language is important. If you see a cat and you call it a duck and you keep calling it a duck, it's still a fricking cat. It's never a duck, no matter what you do. So the fact that they came out and they had this term model, which was a blob of code, it's done so much damage in the marketplace, in the data world. And then when we start moving towards self-service democratization, we know that we lose rigor and that's okay. We're getting a whole lot of new people who don't know how to do it in the way that we were trained. And therefore we gotta help them apply this balance of self-service and rigor without making it awful for them. So what we see in DBT world is they've now realized that 5,000 blobs of code is probably a bad thing. We could have told you that, but that's okay. You have speed to market on your first 100 and that's good. So what do we do? We go, oh, you need to model. And for some reason, everybody said dimensional modeling is the way to go. But if we look at dimensionally modeled and star schemas, what was it there for? It was there because our technology was so constrained. We couldn't just dump all the data in one database and ask a question. We had to make choices of how to optimize it, and the star schemas was a brilliant way of optimizing it. People keep telling me, "Oh, analysts really love star schemas and understand it," and I call bullshit. Yes, we can teach them that a dim and a fact and the grain of the fact and you join back to the dim, and then you've got to do all these horrible joins. But actually, in my view, you go ask an analyst what they want, one big table. Just give me a table of the data. And they go, oh, but if it's customer orders, you can't count customer. You go, count distinct customer, and you get the answer, oh, they won't know that. And then I argue, so you're telling me that actually you can teach them how to join fax and STD type two dimensions and get the right answer, and actually somehow count distinct when you've got a table that's got multiple orders for a customer is more challenging? I call bullshit again. The next thing is we know The first question we always get is a simple one. The first question we ever get is how many? And we know that's not actually the question. That is an exploratory process where somebody says, give me the answer to that. And then I'll know what the next question to answer is. And it's typically how many or how much, yeah, how many orders do we have or what was the total order value? Then we have some kind of breakdown. Where did it happen? Which store? Which product type? Which order values above 100? So we get these wheres and these whys, and then we're going to get the how longs. Oh, how long was it between them placing the order and us shipping? How long was it between them placing the order and paying? And then the exceptions. Oh, how many people placed an order and paid and we didn't ship? Actually, how many people placed the order and we shipped and they didn't pay? Because that should never happen, but oh, it happens in the data. And then we get to the last one, which is why. Why did they place the order and not pay for it? Yeah, and so that story of answering those questions is repeatable. We see it time and time again because the person doesn't actually know what the real question they want to ask is yet. They're using data to explore the problem, get that question, and then they want to answer it. So yeah, I reinforce the messaging you're giving, right? Is that it's a problem for a long time.

Yeah, well, I want to add two more things that happens after, in between the why, which is key, is that part of the why is they have hypotheses. And the stakeholder hypothesizes, doesn't care about the order. So they say, did that person receive an email campaign? What was the last campaign they received? Did they come to our website? Maybe they submit a ticket between when they got shipped and I wanna know what happened there. We all can agree that's the flow. Now, when you look at how we answer those questions, we treat every step like a production system. So if we were gonna write this whole query to do it, well, now you wanna add another concept to that query. Oh my God, that is super complicated because now you're combining multiple and multiple systems. That query gets really long. And because the way that SQL is stack-based, which means that SQL is joined, any sort of thing you're not careful with, you can duplicate rows, you can drop rows, and you can cascade and blow up your whole thing. So I think that process of how we answer that should be the way that we're talking about it. I ask that question, I answer it. I ask that question, I answer it. And it should feel like that kind of minutes. But today we have to go through a whole fucking cycle, what, 20 questions that we know we're going to ask in that order. It would take a data team 2 weeks per question in a whole cycle and materializing table and now have to maintain 50 tables and that shit show. So that's one thing. The second thing that I want to touch upon that you also mentioned is humans and words. Words are so important. We used to have this joke, which is people say, okay, how many sales did I get? Well, it turns out that if you asked the Sales team, how many sales they get, they care about the timestamp being when the customer signed the contract. If you ask the entire executive team, they care about when the customer started their reservation. If you ask the finance team, they care about when the customer made their first payment. These things are different. So just saying, how many sales did we get? It's really unhelpful. And what we've done is we've expected so much information to exist in a column, in 3 words of a column. So we see this table name, Sales and Web Data, and you're like, cool. And it has a column, total sales, but you don't know what that total sales actually really means because that total sales can be based on the first touch attribution, based on the last touch, based on when the sale happened. It could be so many things. And that just language issue is such a fundamental issue. So that's the first language issue. Second language issue that happens is that the person who's asking a question is asking it exactly the way you did. Which is, did they call? Okay, how long did it take them for the next order? They're describing it in a very human-centric way. They're talking about building blocks or like actions that the customer is doing and relating those actions. The data engineer or data person that has to answer it has to take that question from action and time and relating it in this language in English and convert it into SQL, which is stack, table, and how do I join it? That was the first insight that led to activity schema. I said, what if we don't have to change it? What if we can actually take a structure that represents the way people talk about data? And what if we can actually relate data in the same way that people think about relating data? Those two aspects is what made the Activity Schema what it is. The first concept is that you break your world into activities, which is they completed, order got shipped, they got a call, they submitted a ticket, they did this thing. The second thing is that you relate data in the activity schema world using what's called temporal joins. In between the order, did they call us? Give me the last time before they called us what the ticket was. Notice how in those worlds you never predefine any joins. You never have to worry about which system it came from. And it's super analogous to the way that humans are asking questions. And by creating a structure where you can combine data that way and ask and answer questions that way, it just mitigates that cycle. So you don't have to worry about the foreign key not existing. You don't have to worry about weird joins. You don't have to worry about duplicating rows. You don't have to worry about dropping rows. You can add it. And then when you want to add something, because activities can always be appended, you can add many temporal joins without ever changing the number of rows in the data. Once they go, okay, I added the in-between, did they get a ship? Okay, did it get returned? Okay, wait, did before they submitted the order, did they submit a ticket? I can always add as many more dimensions as people ask follow-up questions and slice and dice that data. Without ever having to worry about going back to the original definitions of models. As long as a concept is defined as an activity, you can just use it and be happy.

So let's unpack that a little bit. I'm gonna play it back to you the way I understand it. Tell me when I'm wrong. So you use the term activity. I could use the term event, or I could use the term core business process, or in Data Vault terms, I could use the term link, or in dimensional modeling, it's like a fact. Again, the example I always use, customer orders product. Customer pays for order, or do they pay for product? Good question. Store ships order, or do they ship a product? Good question. A customer returns product to be, return the order. So those are 4 core business processes or 4 events that we'll see in the data relationships and concepts. And so for activity schema, what we're talking about is the term we use is activity. And we say we saw that activity happen from my podcast with Hans. We talk about Peter the Fly. If I was sitting on a wall and I actually watched what happened, With the humans, I see a customer and I see them order their product. And so what we do is we rack and stack those activities into a single table. So now we've only got one place to go. You want to ask a question, that's the table for you. One table to rule them all. And then the second thing is we know that often the question, as we talked about, is the why. What's the time between these activities happening is actually the hard question. And why is it hard? Normally in a dimensional world, we've got to go away and we've got to grab multiple fact tables, multiple grains, slam them together, and then be able to do the time series query about temporal. When did that happen? When did that happen? What's the difference in time between those two? Or in Data Vault, we've gotta grab the links, we've gotta slam them together, and we've gotta say, I've got multiple events now or activities, and then the same kind of thing. So with activity schema, by putting the activities in a single table and having some core metadata effectively around business effective dates and when that thing happened, we can now just write a windowing function. It's the same windowing function to say, take this activity, customer ordered product and take this activity, customer made payment and tell me the time between because we know the structure of the table and that query becomes repeatable because it's exactly the same pattern of query. Did I get that right?

So 95%. So the activity schema way is instead of thinking about it as window functions, we like to call them temporal joins and we found there's only 10 of them. So why are we not using window functions? Because as you add data in and you wanna add columns, you wanna be able to pull more nuanced relationships. So the temporal joins that we have is either first-ever, last-ever. So give me everyone's order and gimme the first web visit they came from. And so I can see the ad source from it. Easy. Gimme the last order they have so I can know if they're an active customer or not. So that's first-ever, last-ever. We also have first-before, last-before. Most common. When they had this order, give me the last time before, and you could add within 30 minutes, did they submit a ticket? Then there's in between, which is the most powerful one. Okay, now I know in between orders, did they email? So not just not ever after, but in between the specific interweave the orders, give me, did it do it? So by having these very specific ways of actually doing it, you end up creating a more consistent standardized way of relating data, which is these 10 temporal joins, and you can add as many more dimensions relating the data to the original starting, what we call the cohort activity. So you start from an order and you can add as many things— last before this, first after this, in between this, aggregate all the orders over, aggregate all ever so I can have the LTV, all the sorts of things, add the next 30 days so I can get LTV for the next 30 days. All that becomes really easy because you're always stacking that data or appending that data to it. So that's the, it just creates a more standard way of relating the data. And you're always trying to describe that customer journey. I care about when this happens and when this happens in between, or when this happens or this happens after, or that sort of thing. Okay.

So if I don't use the word window function, let me give you another pattern that we had years ago. So we used to have OLAP cubes back in the days when again, it was constraint-based. Our relational databases couldn't hold high-volume aggregate queries. So we then had another database, which which was multi-dimensional, so rollup, molep, holep, a whole lot of different techniques. And the goal of that was to be able to get aggregate queries faster than relational databases. And we typically had analytical functions. So what you're saying is, with the activity schema, we get 10 patterns, 10 query patterns, 10 analytical functions that are time-based. So our ability to ask questions over time for those activities or events, and the relationship between them gives us the answer. Cool. So did I get that right?

Yep, perfect.

Excellent. Okay, activities are events. We rack and stack them into a table so we know that there's one place to go to see all the events that relate to something. We have these 10 query patterns, these 10 analytical functions, these 10 temporal things that say we can ask those 10 questions and they always will give us the right answer. And they're the top 10 that we always use. And they're quite constricting if you have to write them by hand. And then the last thing we want to talk about is concepts. And the way I describe it is customer orders product. I have a concept of customer, concept of order, concept of a product. Customer pays for order. I now have a concept of a customer, concept of a payment, concept of an order. So you could think of concepts as a dim and, and star schema world, maybe. Typically, it's something you want to count. It's something you want to manage. You manage your customers, you manage your suppliers, you manage your orders, you manage your employees. So in activity schema, where do the concepts live? Where do the things that bind that activity together to say, these things had a relationship at that point in time.

So that's another uniqueness about activity schema is you actually relate everything to the core actor or the entity that you care about. So you might have multiple entities and multiple activity schemas. So based on your business model, we are a multi-sided platform. We have a person stream and a company stream because a company does things like pay, makes payments and stuff, but a person might log into a portal. So for the e-com example that you used, One of the things you said is you already described it in the language, right? So the customer might complete order. We might ship an order. We also might have another activity called ship product. We might have another activity called purchase product. The customer is doing these behavioral things that we call activities, and that's where everything lives. So everything that you will tie, as long as it's related to the actor, a customer opens an email, a customer has their order delivered, whatever you wanna do, if someone is doing it, you keep it as activities. And the reason why we do that is explicitness. Instead of having concepts and you might have payments and payment, it's like payments exist as an object. We don't like to think about objects in the activity scheme world. We like to think about customer behavior. So a customer makes a payment, a customer has a payment get canceled, a customer may submit a billing inquiry. And as long as you keep it that way, it matches very close to how people are asking and answering questions. And that's what makes it really simple is that if someone has a new concept that they're asking questions about, you can make it an activity. It doesn't really matter. Activities can overlap a little bit. It's okay. You might have, like you said, signed contract, started subscription, made invoice, and you can always ask questions using those words. You can also have some sort of metadata that says this payment ties to this specific invoice. But for the customer's perspective, you can say, give me every time the customer had an invoice and give me the first time after they had that invoice where then the metadata, you can do the metadata additional join where it is the same ID. But for most things, you don't need to worry about that because it ends up being very less common.

All right. Let me just replay that back to you to make sure I got the pattern and to reinforce it. And then I'm going to go into some of the complex ones where I think Activity Schema's got some problems. Not problems, where the context doesn't fit. Be clear here. Every modeling technique we have works really well in certain contexts and these other contexts where we, we are the cost. So, you know, with Data Vault, it's the complexity of the tables that we create, because like you said, We create a shit ton of them. And if you have to query them in this raw form as an analyst, you're screwed. We have to do some extra work to solve that problem. But let's go back to the core of that. So what we're saying is that the activity schema is bound by a core concept. So we might say we have an activity schema. Do you call them tables or do you like—

Yeah, we can call them activity streams. So your activity schema, you have multiple streams.

Okay. But effectively, In my head, it's a table. So we have a table.

It's a single table, yeah. Cool.

All right. And so we have one that's bound by the core concept of customer. So every activity that relates or every event that relates to a customer is in that table. Now, what's interesting about that is we have to alter our language slightly when we talk about who does what, right? So we go customer orders product, customer pays for order. That's good because we start off with the word customer. It's an activity that customer is doing. So we go naturally customer did that. But when I go to store ships product, It's normally store ships product to customer. So in a natural language, customer's at the end, but actually in activity schema, what we say is customer has product shipped to them from the store, because we're binding it back to that core concept. So our language is, it's the same language, but it's slightly changed. We always put customer in the front. And if we can do that, customer returns product, then we have a boundary for that activity, the activity schema.

So just to add one more note on that's really critical. Makes asking and answering the right question really easy. Can I give one example that I like to give? I had a customer, they have an online platform and they were trying to understand when they launch a class, how many people sign up. So they wanna know what's the best time to launch a class. They had done this analysis and they were great. We have a table, it's classes get launched and then they get whatever the average number of signups. And they say based on when the class gets launched, what's the average number of signups? Makes sense? So in Activity Schema, that doesn't let you do that. It's like, 'cause like you can't, you, what do you mean when a class gets launched? It's, but that's important because when a class gets launched cannot impact the people signing up because if it gets launched at 6:00 AM and everyone gets an email at 8:00 AM, you care about when people actually see the class. Even though you're saying based on when I launched the class, how does it impact things? What you're actually saying is when a customer first sees a class, How does that impact the likelihood for them to sign up for it? So based on when customers see classes, and that's when having that core entity makes asking those questions really nice because we saw a lot of bad decisions where people were— this whole story why this customer ended up buying Narrators. After we did this analysis, we realized that, oh, it turns out when we launched a class had no impact, even though it looked like it, it's because it's actually based on when we send the email, because that's when everyone actually sees the class. In the activity schema world, that would have never happened. They would never been able to make that mistake because you only tie things together through the customer. And therefore it's when the customer sees the class, not when a database updates something. And that's a really important part that has led to a lot better analysis in the activity schema world.

If you come back to Laurence Kors' model around who does what with BEAM, and it's the same thing. We go, who does what? So we'd see employee launches class. There's no customer in there. There's no customer in that, in that who does what. So then it might be system notifies customer that course has been launched, something, something horrible like that. Okay. So we bind the activity schema table to a core concept. So if I use again, my customer orders product, and we look at an organization that actually manufactures the product before they send it out, we're gonna see another activity schema table, which is bound to the concept of product because we're gonna see maybe products being manufactured, products put in warehouse, products shipped out to customers. So we're gonna see those activities because we know on the fly, we're watching that product and we're watching what happens to fulfill the order, all the activities or steps that happen to that product before that process is finished. Did I get that right? So we end up with a bunch of core activity schemas. And what we know is actually if we look at an organization that's modeled well over time, there is probably somewhere between 7 and 20 core concepts, core concepts in that organization, in a really complex organization. And we also know that actually customer is always the primary concept. If you're, if you're an organization that deals with people, you might call it customer, stakeholder, citizen, whatever, but a person or an organization, and we get some complexities there, is always the driver of most of the events or the activities. Is that right?

Yep, exactly. So in Narrator world, you would have a product stream and a customer stream. They might have some similar activities, like a customer might have their product manufactured And a product stream might have started manufacturing. And the reason why that's really important is when you're asking questions about your operational excellence, you might say, how long does it take for a product to be manufactured? Or how many times is a product moved in a warehouse? And based on those questions, you'll go to the— based on the entity you're trying to change, you will go to that stream. So if you're trying to change the manufacturing processes, you would look at the product stream. If you're trying to impact the customer, you would go to the customer stream. It's okay for activities to be in both streams. Cause a customer's product also might get started manufacturing, a product might complete manufacturing. So many things that happen.

Okay. So there's another one there that I missed. We all know that often we get given a question where there's no data for it. Why are our highest value customers buying that product at 2 AM in the morning? We just, we don't have data for that. We don't know. Good question. Let's go get some data. So that, that example that you used, that product, so customer has product manufactured for them, cause we're binding this one to the customer activities. Often that's not available. That's not data we store. We don't actually say when we're manufacturing a product, unless it's a product on demand for that customer, we typically manufacture a bunch of products, we put them in a warehouse, and then we pick them and send them. Amazon doesn't say this book was manufactured for Shane Gibson. It's in the warehouse and they're picking for it. But if we can actually say that product was manufactured for a customer, if that's the way our core business processes work, then we inject it both into the product activity schema and into the customer activity schema, because why not? There's no cost. And now the ability to query that is much easier because we don't have to cross-join across those two activity schemas, which is where I got to, was my next question is actually joining those activity schemas must be a nightmare because whenever we do any joins of any data, that's at a different graph. Okay, so you don't.

You don't, and you never need to because you shouldn't ever. So like, you see this a lot. Let's go through some examples. Can you come up with an example where you might think you might want to join two streams?

Oh, let's do the standard one, which is I have a prospect coming into the website and I don't know who they are, but I have some anonymous prospect ID because it's Google. Then they come in and then they sign up for a form. And so we convert them to somebody we know via an email address. And then we'll go out and we'll ping them with a marketing campaign. So now I combine that to my customer. And then they come in and buy something. And then we ship it and then we do fraud detection on them.

So far everything you said is still related to a person. And in Narrator there is a concept of a customer and what's called anonymous customer ID, which is if you don't know, you have your global identifier, which let's say it's an email and they might be a credit card number, they might be a name, last name, zip code, they might be a cookie, they might be a prospect ID. So it's still the same person. But taking your example one step further, let's say then they sign up as a company and now they're a company or an account, and an account pays an invoice. And you wanna know from that person who first signed up, do they pay? How long take them for that first person to pay? In that situation, which we do a lot, you might have the activity, the person's Ahmed who signed up. Ahmed might have Ahmed company pays bill. So you might have an activity called company pays bill. The same goes with the company, the account stream or the company stream. We might have employee received email or employee submitted lead, or we can have those concepts from the perspective of the entity you're trying to change, which makes it super easy to query because when you're asking questions, again, your questions are often, I wanna know, I'm like, does that person who comes in, how long till their company makes a payment? Well, great. I have company makes payment, submits a lead, and it's in the same stream. Super easy. Everything is nice and clean. And yes, when I'm building those activities, I might in my backend database have company and company has users and I might join on the company user and I'll just add an activity for every person in that company at that time that they made a payment, it's okay because from the perspective of that person's individual stream, it makes sense. And that's why we built this thing on top of a warehouse. You mentioned activity versus event. I use the unique word so people don't get mistaken for the way that you naturally treat CDPs where you have to log the events. No, you're in a database. You can Combined, you can create that company pace. You can put that on every single person's dream and on every person, who cares? You can do that and it makes it really easy to answer those questions.

Okay, so that's key though, is that where we see an event come through and using other modeling techniques, we'd ask the who does what, where we say in that event, in that core business process, what concepts are involved? So customer orders product, we have a concept of customer, concept of order, concept of product. We bind that together, right? Because it makes sense because we know we're going to ask that question. What an activity schema, what we do is we may end up with a customer activity schema, an order activity schema, and a product activity schema. And we would push that one activity to all three of those schemas because they're bound to it. And now we can ask the question between. Okay. I get reading it. I hadn't picked that up.

Yeah. And just to make sure, like, it's really rare that you actually have an order activity stream or a product stream. It's only when we work with Costco-like companies, they have a manufacturing process where they're asking questions about the manufacturing and they're asking questions about the person. For a typical e-com, it's usually just that person stream because that's all you really care about. Everything you're asking is from the perspective of a person and you wanna know like, give me the outliers of when people take the longest to get their product delivered. It's all relative to the person. Actually, you don't give a shit about the products. You care about the person who's waiting for a long time.

Yeah, but what we know is when we look at organizations, we focus on administration events. We focus on accounts clerk entered invoice, accounts manager approved invoice, somebody released invoice. We focus on those subprocesses. Because that's what we see people doing. And if I, and again, let me just play this one back to you. So I'll use a different example, insurance company.

Adam. Yeah.

A customer applies for a cover and then we know there's an actuarial process actually going through. But effectively, when we think about it as a core business process, we always bind it to the policy. Customer applies for cover, we create some form of policy, we then get the actuarial team to go and look and say, are we going to approve that policy or not? And what we don't tend to do is we don't tend to end that off with the customer name. We tend to divorce them from that activity, even though it relates it. So all we are saying is actually customer has policy reviewed by actuarial team.

Customer's policy gets updated. Yeah. Yeah.

And we know that policy relates to them.

Yeah. Yeah. And then you can answer really complex questions really easy. So for example, as like policy gets updated, you might want to know every time the customer is meeting with the sales team, the customer starts an opportunity, the customer has a meeting. Customer might go into discovery, a customer might have another meeting, and I want to know when that they had that meeting, what was the policy revenue, for example. This is really easy to do in the activity schema because you say when a customer had that meeting, what was their last before updated policy, and I can then see the revenue there. So even though the policy might be going through a separate experience and a customer is going through like a separate sales experience because I have a customer start opportunity, customer enter discovery, customer had a meeting, customer had their policy updated. I can always say, give me the last before policy updated. And now I have this nice table that makes that really complicated question really easy.

And then again, we'd probably see that in the type of industry you're in would drive what your core concept is. So we're doing some work at the moment in the utility market, and it's not about customer at all. It's about asset, physical device. And so that's, that I can infer there that actually the majority of their activities will be bound to that one schema. I might have some secondary schemas, but actually everything we talk about is around these assets, this bit of equipment. We don't talk about customers at all because it's not for that part of the business. It's irrelevant.

We had buildings and different things like my building gets acquired or building goes for review or building gets start opens and all sorts of stuff.

As we're building out more and more customers on our platform, we're actually looking at the metadata because we're really intrigued by this question. Based on our consulting experience, we have this gut feel that from a who does what, from a concept point of view, that most organizations when well modeled have somewhere between 3 to 7, maybe a maximum of 20 core concepts if they're a really complex business with actual multiple businesses embedded within them, multiple unique businesses under the same umbrella. So from your point of view, how many activity schemas does an organization typically have? They have a core one, which is customer, and then what, they have normally a 3, 5, 20, what's the rule of thumb for how many others you've got?

I think most companies have one. I think probably 20 or 30% have two. I think the maximum I've ever seen in all the customers has been four.

Okay.

It's really rare that you have more than four, unless you have multiple product lines. I would say every product line usually has something they're trying to change. And it's really rare that you have so many things you want to change. Even WeWork, which had many product lines, there's people, there was companies, there was people company. Building, and that was like most of things. And then maybe have internal stream, which is like an employee gets hired, employee, that's a computer, an employee gets for all the people analytics. Do you have other things? Maybe. It's just less common. And the point of it is not to really try to think about it. Like the whole point of activity schema is you don't have to decide those things now. You can always add another activity. The whole point of it on warehouse, you can always, one activity can never be used and you can just delete it. You can always add another stream. I would like to always say is start with the questions you're asking, see what the core entity is. You'll just keep going. And then if you keep feeling like you're constantly trying to add new data, add an activity, it's fine. And if you've created an activity that is never used, just remove it and it's fine. And because everything is on top of the warehouse, we're just really remodeling the data in the warehouse. You can always add and remove and change your mind. Your data's not lost, it's just captured in the warehouse. So you could just kind of go in and ask and answer questions.

Okay, so let's just replay that back. What we see with any modeling technique is there's always a bunch of anti-patterns. So within Data Vault, we go multi-active SAT, where in the satellite, there's more than one active record. That is an anti-pattern. It causes you a shit ton of problems. Star schemas, if you have multi-grain facts, you're dead. So don't do it. So what you are saying is if you are using activity schema and your organization has 20 activity streams, maybe have a look at what you're doing, cuz you're probably taking an old modeling technique and just trying to replace it with activity schema.

Exactly. Yeah.

The second pattern that in there is this is not one data modeling pattern to rule them all. Right? Because what you're talking about is the data's in a warehouse, and now you're applying activity schema to that data. So you're not saying that you get your data out of your system, the capture, and it goes straight into an activity schema, and that's the only data modeling technique you use. Is that right? You're saying that you may use other modeling techniques if they have value, but the activity schema is the core data modeling technique you use to ask any question. Is that right?

Exactly. So we like to be in the warehouse. You can add dimensional joins on top of the activity schema, of course, if that makes sense for you. It's just exactly that, like it's just the core and it's designed to ask and answer questions. You talked about anti-patterns. We say if you built another, if you build more than 2 streams, you should ask yourself why. If you have more than 30 activities, you should stop and say why. Most companies don't have that many. And then 3, if you find yourself taking more than 10 minutes to answer a question, you just stop and say why. And if you're spending more than 15 minutes to define an activity, you should stop and ask why. Literally, these are alerts that we have in our product. Where if the user is opening the transformation editor and they're spending more than 15 minutes to define an activity, we will put an alert. This is usually— this query should be simple. It should be really slight mapping, not really complicated things. And then everything else happens in the activity schema that makes it really easy. So you don't have to worry about it.

That's good. 'Cause again, when I looked at it, I thought for me, Data Vault is the middle combining layer where we combine data from multiple source systems into one place. Around those concepts made sense to me with the Activity Schema on top, where I'm effectively making it incredibly easy to query that beast of a model. But I'm still gonna think about it a bit more, 'cause I'm not sure why we need the complexity in the middle. I need to go through that.

Most of our customers don't do that. Most customers go production systems, build activities on top of them, and then go and answer. And it's really, it becomes really nifty 'cause you, each activity is often one concept. If you are, if an activity has more than one concept, then you're like, well, that should be two separate activities. Stop doing that. So it makes it really, it's, it's, it should be like a trivial mapping that you put to that. You're just restructuring your data into the model. Okay.

So again, if we think about core concepts and who does what in those business processes, like I said, there's always 3 to 7 concepts involved in a relationship when we watch it happen. And again, Andy Patton, you get to 20 things in that relationship and then you've got a, you're modeling wrong. So if we think about that. One of the things that I struggled with was, yes, I hold the core concept of customer or person against that activity stream, but I naturally want to add the other concepts in. I want to do group buys. I want to report by those things. So I naturally want to bind it to there was a product and there was an order against that activity event. And the activity schema was limited to, I think, 3. And then I remember you came out with somebody, I think it was you, came out with something that said you've gone into some larger customers and, you know, that didn't scale because we know that once we get these modeling techniques, we then get all the edge cases where it's like, okay, 9 times out of 10 it works, but if you happen to be doing that one thing as a company, then we do it. And so talk me through that, these secondary concepts that are bound to that activity.

Yeah, so the way we designed it was we said, okay, most activities have 3 features, so we'll just maintain 3 features. And then we said if you wanted to more than 3 features, you would just do a normal dimensional join to add the additional features, just traditional. So those 3 features really— I wouldn't— this was designed 7 years ago. Back then, JSON and unstructured data wasn't really common. People didn't put that in their warehouse. So I was like, okay, 3 features for most things, it's really fast to query, and then a join when you need to do additional is fine. The honest answer is people got really obsessed with the 3 features, and then what if I want more? And I'm like, you don't really need more, and you have dimensional tables. Like, I can tell you how many companies Out of the activities, it was 4% of our activities had dimensional joins. Most people didn't need more than 3. So I was like, okay, you still have your dimensional joins to use and we support it, but you don't really need it. But people were really upset. So we're like, great, let's just make it a JSON blob so you can add as many features as you want. Part of that is you talked about anti-patterns. We do a lot of checking for this and recommendations and our whole system is designed to help them avoid this. But an order, people, I wanna know the ad source of the order and I wanna know the attribution of the order. You're like, no. Those are derived. Those are shoving things in features. An order happens. What is related to the order only? But the ad source where we acquired the customer, great. First ad, first ever started session, grab the ad source. Don't shove that into features. And people have a tendency to create hundreds of features and you don't need that. You only need the features that are unique to the order because you can always borrow features from other activities. So if you want to know how many people often will do, they'll have, I don't know, there'll be a lot of things that you might think you need, but you don't. You want to know when a customer purchased the product, they don't want to know how many inventory we had of that product. Or that might make more sense to put it as when the customer views the product. So you might say, give me last view. So we allow you to now have a JSON blob where you can add as many features as you want. We try to recommend that you don't put more than 3 or 5. We still support dimensional joins. You can still do that, which is helpful if you have a table that you just wanna link to, but we try to avoid it. We still give you the flexibility to add as many as you want with the JSON now and the structured data, but we try to recommend that you don't shove like a million features.

All right, so let's unpack that cuz there was two patterns in there. So the first pattern was I've got somebody coming through to the website doing a page view and I want to attribute where they came from. And a natural reaction is to store the attribution of which ad they came from. Against that event. But what you are saying is it's an event in itself. Customer viewed ad and then customer came through and viewed webpage. And now we know they viewed the ad at this time and they viewed the webpage at this time, and therefore those events activities are correlated. So we, again, we denormalize the attribute back to being a core activity itself. And then, so that's the first pattern. And again, if you've been trained in any other type of modeling technique, you, you Your brain works as they're just attributes of something. They're not core things. So you've got to untrain yourself and go, that attribute's actually an event, it's an activity. Okay. And then the second thing is, which was my next question, is in the way we talk about core concepts, customer orders and products, and we talk about detail. The customer has a name, the order has a value, the product has a SKU. And so that detail is important. So what you're saying is in the original pattern, you would have 3 features. And to get the detail of the customer, you would join out and grab that detail from another table that describes the customer name. To make it more flexible, because we hate constraints, whenever we see a constraint, we think we're losing something and we focus on, oh, I can't do that. What we do now is we effectively land those concepts as a JSON object. And so the pattern technically is pass the JSON object, get that core concept, and then bounce out as a join to the detail table to say, okay, that thing has a whole lot of attributes or detail about it that relates to that concept within that activity. So yes, we can answer that question out of that activity schema really easily, but there's still a little bit of complexity because every modeling technique has complexity. And so this one is when you want the details, you've then got to go and go out and get the details from somewhere. Okay. That makes sense.

Yeah. So yeah, you have to open the JSON blob and get the details from the JSON blob. So the activity schema is, it's 90% of the stuff you go to, but there's also what we call our customer dimension, which just has the name and the age and all the stuff that you put on the customer. You might have aggregate dimension, which is like if you're spending money, like I spend $30,000 on this campaign on this day, and those can be joined on top of that activity schema when you need it. But like most of the stuff you're doing is within the activity schema and you just unpack that JSON and pull whatever features you need from it.

Cool. So one of the things I love about Datavault is because there's so few objects, there's so little code for every, for 90% of the cases. So we talk about create hub, create set, create link, where we create those tables, load hub, load set, load link. It's only really 6 bits of code. And the benefit of that is we harden that code. We use that code every day. That code gets immutable. We eventually, we stop playing with it. And so what I can feel about activity schema is that's the same when we breach out to those detail tables, those what you call dimensions, is that we know it's pass the JSON field, find that feature. Look that feature table up because the naming should be the same. If I make the name of that table the same as the name of the feature value, the feature, now I can automate it.

It's embedded in JSON. So it's just like unpack JSON. We actually just literally store a JSON blob called feature JSON on that table. So in that row, you just unpack it. Yeah.

So if my features are named the same as the dimensional table that holds the detail, I just now have one bit of code, which is unpack that JSON, go look up that table, get the detail. If I was writing it myself.

Oh, the details would be part of the JSON. So you don't have to look it up. All the details are in the JSON blob. So the customer dimension, well, you join on customer, that's like standard dimension. And then on the activity, it's like the order details would just be in the feature JSON on that row.

So order value would be in the JSON?

Yeah.

Ah, okay.

So you don't have to look at it, you just literally unpack it. It's, it's right there.

Okay. But you're doing it as a nested JSON object. So you're effectively embedding detailed tables for each of those features in that JSON object.

Single dimensional. You never make that JSON object have an array. So within that order, so if you're used to Snowflake, you would just say word activities order feature.json.order value, and you'll get the order value feature.json.tax amount, and that gives you a tax amount. And it's like that. So it's really easy to use. We don't let you double nest that. It just makes it unnecessary.

So if I play it back, effectively you're taking the dimensional attributes and you're binding them in as a JSON string against that activity in the activity table. So I'm not going out to another table. And then how do we deal with change then? So customer changes their name.

Yeah, great questions. And now we're like in the realm of the paradigm and then how you work around some of these things. There's two ways of doing it that we support. One thing is that if someone is, if it matters to track change or it doesn't matter. So let's say you, so remember you have a customer has an attribute table and you might just update that table's often materialized viewed and That table might just have the updated name, but let's say if customer changes something that's important for us to track. So you might have that as an activity. So customer updated their location. In the Narrator world, which is activity schema with the edge cases, we do support what's called a slowly changing dimension where you can say, okay, I have this other table that's customer credit card and when updated at, and Narrator can take that and then do in the background for you. A last before join and bring that into and make it look like a dimension. So there is those additional cases, but yeah, in the pure activity schema way, you would just do like, give me my customer's activity updated location.

As I said, with every data modeling technique, there's the core things it does really well, and then there's the edge cases that always bite us in the bum, and the key is understanding those and dealing with them. Okay, look, this has been awesome. We're pretty much running out of time. So I've got a couple of quickfire questions.

Woo!

So I haven't done this before, so this is going to be fun. There must be a standard type or a number of events that you see on a regular basis for specific use cases, specific industries.

Yeah.

Cool. Do you publish those openly?

Yeah, so we are, we're beginning to do that with now for e-com and we have that as what we call packages. And in next year we're going to actually open source. So you'll see it in our doc site, docs.naira.ai. We give you the common activities per industry. We also have realized that people just like starters. So we— if you're an e-com company, you come into Narrator and you have Shopify, it'll spin up 7 activities for you and you can go in and edit them. We also do that for spinning up dashboards and analyses that are just common that you do. So idea of Narrator is you get started and we would spin off those activities, those dashboards and analyses, and we call that a package. So that's an e-com package. And starting next year, Narrator is going to actually allow all our current customers to upload their own packages or own analyses or own dashboards. In a world where everything is on top of a standard structure, people can actually begin sharing their work. Even though you might be— you're two e-com companies, you still have completed orders, shipped products, we can talk about it and we can do analysis on it, but then when we go to implement those analyses, we often have to deal with everyone's unique data. But in the activity schema world, everything is the same. Even though you have unique definitions for what an order is, everything else for the analysis is the same. We're actually allowing our customers to begin sharing those things. And buying and selling them across companies and just make that really easy. Excellent.

And then we all know that we start off with one data engineer if we're a small company, and the standard joke I have is I've never seen a one-person data team. So we always go to 3 to 5, create a team, and then we see some success and some value, but we get this massive backlog. So we then scale and we go to 3 teams, right? And then we have to figure out how to break the teams up. Is we gonna do it pipelining where it's an awful waterfall process of requirements handed over to build, handed over to QA, or are we gonna domain-bound them? So we go, you are the finance team and you are the marketing team. So activity schema, if that's the core modeling technique we are, we are using, that's gonna drive a lot of our scaling patterns.

Yeah, it's like you said, we don't see one or two-person data teams. I see a lot of one and two-person data teams. I actually have seen many companies, our biggest leads come from companies who just laid off half their data team. That's where usually they get interested in Active Schema or Narrator. And usually because again, your definitions are so simple and most of the time you're just reassembling those definitions, we often see answering question speed is 10 minutes. A single data analyst or two, a team of two can handle 100 questions a month, which is unheard of. It's just by having a simpler data modeling paradigm, so many things go away. And so we see so many smaller teams doing a lot more work, a lot of work. When Narrator had a consultancy arm, we used to have one data analyst per 10 companies. That was the ratio that we were doing. And you can see that value in just that repetitiveness and the reusability of a tool, of the activity schema.

If we were going to scale the teams or we're going to distribute them out into the business because we wanted to go to more decentralized distributed model, what we'd do is we'd do it around the activity, the core concept of the activity. So we'd have a team that looked after customer activities. And we might have a team that look after manufacturing activities.

Exactly.

And then the key handoffs would be when the manufacturing activities were defined and they had customer in the sentence, then they'd notify the customer activity team, hey, here's another event that you need to do. Can we push it for you? Or do you want to go push it yourself? So that's the shared languages where your activity actually has one of the core concepts for another activity team. You just let them know that you've got this thing that's useful for them.

Okay. Yeah. Also, it's very common for you to have, because it's a SQL and these are actual tables in your warehouse, you can just literally write a SQL query that wraps around the other stream and brings the data in. That's really normal too.

And so one of the other ones that I'm always intrigued on is this pattern of data mesh, which I was really excited about in the beginning, 'cause I thought it was actually gonna change the way we work. For me, everybody's just buzzwashing it now. But if we were gonna mesh this, if we were gonna push this back to the software engineering team to do the data work without us, This would be helpful, wouldn't it? Because effectively, once you taught that software engineering team what the activity schema is, they're just gonna actually populate the activity schema table out of the software that they do, because they know they're building an e-commerce website that relates to customer, and they know what the events are. So we should be able to teach them to use this technique and have them do the work without us, shouldn't we?

The answer is yes and no. Like, in theory, that sounds great. In practice, those people are really bad at defining activities. So the reason why we shifted away from people, like, streaming events in. We always get that, why didn't we just build a CDP with all this extra power? It's a super powerful CDP. And the reason is that software engineers have bad practice of track everything. They just want to log open modal, da da da da da, and then you end up creating so many garbage activities that are so nuanced that no one uses them. So I actually think about it backwards, which is if you have a single data analyst answering your questions, as you get questions, start defining activities only as questions arise, and then you'll see what activities you actually need, because so much of that data can exist in your database already. Most of people are asking about primary database things, so you already have that data and you can build those activities and they end up reflecting the concept. An example is WeWork didn't have in our entire database systems the concept of an office or a desk, and that's what we sell. But WeWork, the engineering team thought about it as reservables and reservations and da-da-da, The whole point is to make activities as closely resemble the way that customers are asking questions, not how they're represented. So we are very much for don't start from what you built and say, let me model it. Start from what people are asking and build models that represent that language. And therefore, because you're going from the stakeholder's perspective often, I don't know who you call that on your shoulder observing everything, that observing person doesn't give a shit what's in your database and what your engineering is calling it. So I would lean toward avoiding that. If you have an org where the software engineers are super tied to the business and they're really understanding, then it makes sense. We do have CTOs of small companies that use Narrator every day. The flexibility of building business objects so they don't have to query, building more stakeholder representative activities, and then answering questions that way.

And last one, ChatGPT, because we can't do any podcast now about ChatGPT.

Yeah.

So we're seeing that everybody's runway's gone unless you're a generative AI product. We see every incumbent and every new startup doing text-to-SQL.

Yeah.

And so one of the interesting things for me is whenever you watch those demos, you sit there and you go, that is some lovely curated data. Yeah, exactly.

Exactly.

And we know that joins are difficult, and it's going to be interesting to see how any of the generative AI products deal with that. But again, one of the benefits of the activity schema, I could guess, is we've got one big table.

So, exactly.

In theory, ChatGPT will love it.

We do. And there's 3 currently open source projects on top that use ChatGPT on top of Activity Schema that make that really nicely. Narrator's building our own version, which allows you to do a lot more complex questions. Bart Moses, founder and CEO of Monte Carlo, recently wrote an article about the only way to get generative AI to ever become very popular in data is to have it on top of a standard schema. And she references Narrator and Activity Schema. 'Cause that's currently the only real standard. Everyone else has modeling paradigms and they let you build whatever tables you want and you have to think about it and it's up to the engineer or the data engineer who builds it to represent it. Activity schema is like, no, no, no, no, there's rules and there's one way of structuring the data and you're gonna follow it. And because of that rigid structure and the fact that you're only using activities and temporal joins, it has opened the world for a lot of tools like Narrator to exist, but also ChatGPT to be able to— that's just such a controlled space. And the language that the human is asking questions in is modeling the same language that the activity schema uses. So you're using words like in between and first and last and made payment and when the customer called us. And so it makes the activity schema is, I believe, to be the only way that ChatGPT will ever work on top of data. And I think a lot of people will share that belief is that this is a standard structure, and the more people that abide by that standard structure, the more we can share, the more we can reuse, the more we can have ChatGPT on top of it. And I believe in— that's what I believe in the world. That's why I started it. My first ever raising was about sharing and reusing your analytics and your work, and it has to be on top of a clean standard structure that still supports every company to customize their definitions. And I think that TV Schema does that really nicely. It allows you to have full control over definitions. But still have concepts that everyone can use and reuse.

I think I agree with you partially, and then I'm going to disagree with you partially. I agree that the more you get highly opinionated products, the faster and more reusable they become. And then you always got to watch out for the downside, which is by being highly opinionated, you're removing some choice. That has some consequences, but that's okay because you get the benefit of it's faster and it's more bulletproof and all that kind of stuff. And so for me, With ChatGPT, your highly opinionated structure is the schema itself. I think there's another pattern which we'll see a lot of, which is that there's config or metadata, which holds the opinion and the data structures themselves are a little bit more flexible. But then the same outcome, because we now have an opinionated layer that, that holds the context, the semantics, and then hints all that to ChatGPT so it doesn't have to worry. So I think these two implementation approaches of the same pattern, which is standardize things and therefore the engine knows what to ask in a repeatable way every time.

Yeah, 100%. And I think semantic layers, I'm like a very big fan of. And the thing that I would always say, and not in this context, is that if you're a company that's answering a series of questions that you can define a semantic layer on top of, you should do that. Like, you should definitely do that. I think that the things that we notice with especially fast-growing startups is that the questions are always unique. They're just off the top of the head. So sitting and trying to imagine the semantic layer ends up having way too— like we talk about Airbnb's data dictionary and their metrics layer, they have 10,000. At that point, you can't differentiate column 972 and column 322 because the words seem so similar. So at that point, semantic layer is garbage. The activity schema is really good for open-ended fuzzy repetitive questions. Once you build activity schema and you start answering questions and you're doing it, Some of those questions you just want to fucking put in a table and just reuse. We allow you to do that in Narrator, materialize it, put it. The whole point is when you have a new question or you want to add a dimension or you want to slice by this, Narrator makes that really easy and Activity Schema lends itself really nicely to that chaos building with more and more questions. So if you have a clean structure that you want to represent and model, go ahead. I think that's what some of these paradigms help you reuse If you have a lot of new questions and there's fuzziness and there's dynamicism and you're dealing with a chaos situation, do it. So yes, if you're Nike and you have quarterly metrics that you're doing or you have CFO reports that you need to be very specific, don't use Active Schema for that. Those are CFO reports, they need to be done a specific way, do it. But if you're a machine learning team or you're a data analyst or you're at a startup where your company is growing really fast and you have hundreds of data requests, that's when the activities team really lends well because that speed is really important and you don't need that— you need that flexibility and speed just will carry you through victory versus the world of a CFO report where like formatting and organizing and it's very specific. That, build it, use a SQL query, put it in a semantic layer, model it, like do it normal way. It depends on which modeling technique you wanna use. But yeah, for fast questions, iterating, Answering questions really fuzzily, there's nothing better than Activity Schema.

It comes back to that. There are patterns that are brilliant in a certain context, and there are patterns that are harder in other contexts, and there's patterns where actually it's an anti-pattern if that's your context. Most people it doesn't work for. I still say have a go, cuz you might be lucky. But the idea is experiment. And so again, one of the things you said a little bit earlier, and I didn't reinforce it, is this idea that it's an agile modeling technique. You start small, start with Typically customer activity schema, and then add a few events, few activities, and then add a few more. And then if you need to break out to a second stream, do it. And then if it then doesn't have any value, kill it. Yeah. Cat or not pets. That's what we're after in data. Excellent. Hey, look, that's been great. If people want to get ahold of you, how do they find you and talk to you, see what you're writing about, find out what you're doing? What's the best way for people to get in touch?

Yeah, you can add me on LinkedIn. My email is just ahmed@narrator.ai. Feel free to ask, email me any question. If you come up with an edge case that you're like, how does Narrator work in that case? Shoot it. I love those edge cases. And 3, check out Narrator, NarratorData.com. It's where you can experience Narrator, sign up, try it. It's really, it's just, it's designed to be the activity schema plus all the edge cases that we found from working with hundreds of companies. So I'm excited to interact with every single person who sees this.

Excellent. Thanks for coming on the show and I hope everybody has a simply magical day.

Awesome. Thank you. Data Magicians was another Agile Data Podcast.

If you'd like to learn more on applying an agile way of working to your data and analytics Analytics, head over to agiledata.io.