#162 Scaling Data Engineering in Retail with Mohammad Sabah, SVP of Engineering & Data at Thrive Market

Hi, I'm Maggie, VP of Curriculum at datacamp. And I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

To me, really, ownership mentality means somebody who treats this company as their own and really don't look at it as a, let's say a 9 to 5 or 9 to 6. They look at it as, how can I really make impact to the end user? And the end user in this case could be an analyst or it could be a member, like actually a member on Thrive Market, or it could be a stakeholder. So I think that to me is. And if you have that mentality, then you will not look at solving problems in a silo. You will not look at solving problems in a limited fashion. You will look at it holistically.

Welcome to Data Framed. This is Richie. While data science and AI are perhaps the most photogenic data roles because you get to solve business problems and build fun things like chatbots, actually none of your corporate data ambitions will come to fruition unless your employees get the right data in the right form at the right time. Making sure that happens is the job of the data engineering team. So today we're going to have a deep dive into data engineering. As with a lot of the topics we discuss on this show, there are many devils in the details. So I want to get into how data engineering teams work. And today's guest joins us from Thrive Market. So I also want to learn about some of the retail use cases and challenges for data engineering. Mo Sabah is Thrive's Senior Vice President of engineering and data. He's got a particularly fascinating role because as well as being in charge of data engineering, he also runs the data analytics and data science teams. So I really want to pick his brains on how these teams interact. Data engineering is one of those things where it's easy to do simple things on small data sets, but the real magic happens when you try to scale things up. So I also want to quiz Mo on how he's scaled up Thrive's data systems.

Hi, Mo. Welcome to the show.

Thank you, Richie.

So I'd like To get an overview of how you make use of data at Thrive Market.

Absolutely. So I've been here at Thrive Market for the last five years and four months, almost five and a half years. My title is officially SCP of Engineering and Data, which means I run the entire tech org which is engineering and data. Before this I was in a similar role at Honest Company, another e commerce company. And the reason I ended up there was I actually built my own startup in the space of personalization. So we were solving the cold start problem for people who are new to our platform. It would actually learn from data that we have about you and run, recommend you and personalize your experience right off the bat. That company got acquired by Honest Company, so I ended up there and prior to that I was in the Bay Area. So I've been living in LA for the last close to eight and a half, nine years. Before that I was in the Bay Area, I was running engineering and data teams at Workday, prior to that at Facebook and I was a data scientist and engineer at Netflix and Yahoo. So I spent a lot of time in this domain of direct to consumer. And that's a space I really enjoy where you can actually learn from the actual interactions of the customers and learn from the customers and the members directly.

That's cool. And are there any particular goals that Thrive has around using data?

So Thrive is really a health first membership company for conscious living. And our mission is really to make healthy living easy, affordable and accessible to everyone. We operate on a very environmentally sustainable operating model and also a hyper curated catalog. So think of it as a mix of Amazon and Whole Foods. And so you get on this membership similar to what you would do on an Amazon. You pay a yearly fee to access our hyper curated catalog. And also very sustainable. Now data plays a very critical role in all of this, as you can imagine, right? Data forms the very bedrock of everything we do. And if I were to break it down, we use data for making all strategic decision making and we are going to dive into it a bit more in the latter part. But I'll just say if I look at it at a top level, we use data to make decisions, which means we have very strong bar on the data availability, data reliability and quality. The second area would be we use data to personalize the user experience, which actually makes the user experience and the member experience so much better. We also use it. And the third area would be we use it for a bunch of optimizations, right? We would optimize our campaigns, we would optimize our Spend based on what we are learning from the way every dollar is spent and what it does for us. As I said, data forms the very foundation of everything at Thrive.

That's interesting that using data throughout the whole business. I'm curious as to how the data engineering team fits into these data cults across Thrive.

A few things about the way we function at Thrive. We function in the form of squads, which are these cross functional groups. And that really encourages better collaboration, better data standardization and also better reusability across the board. So the data engineers are embedded within these quads so they also learn about the business domain and they can make smarter decision making in terms of what is the best way to surface that data. And that also puts them in a very good position in terms of learning about how the data is used by the rest of the company. So that's the first aspect I'll give you is how we operate on a day to day basis in the form of squads. And these squads could have folks from, let's say from finance for a given squad, or they could have folks from the front end team, the back end team could be operations and so on. The second thing I'll say on this is we have organized the data engineering and data analytics into one organization. And the reason for that is very simple. Data analysts form the majority of the user base of the data that is generated by the data engineers, right? So data engineers are really responsible for the data warehousing for the etl, data reliability, data availability, data quality checks and all that stuff. But really data analysts are the ones who are using that data in a big way. And of course there are rest of the company too. Like there's data scientists, there are applied ML engineers and so on. But the way we have positioned it is we have built the team around data analysts and data engineers working together. So when there is a and it solves multiple problems for us. One is the data engineering team is not siloed into its own, which I've seen in the past in some other companies. That silo is something that hurts you down the road because the data engineers could be building solutions that may not be pragmatic, may not be reusable, and maybe very specific to a certain domain. Having them embedded inside data analysts a encourages a good discussion between the analysts and the data engineer so they can look at the problem holistically and solve it holistically. The second big area is what I highlighted earlier on reusability. We don't want to be reinventing the wheel. So if the user access Patterns exist. One of the things that I would like the data engineers to look at is as they are talking to these analysts, understand what are the common themes that are emerging, what are the common threads, what are the common sources of data that are accessed together. And we can use that to form these core data sets, which are wide and can be accessible and can serve multiple use cases. So that at a high level is the way we operate in data engineering. And we have done it very successfully, which has really. And we've been able to reduce tech debt, we have been able to move fast and also build data sets that can be reused across domains.

That's really interesting. Think about the data analyst being almost a customer for the data engineers, but then you bring them closer together so they can work more efficiently. I really like this idea you mentioned of reusing like patterns, of reusing data sets maybe from one project to the next. Can you give me an example of how this reuse works?

Like, what are these?

Like, I think this is user access patterns. Can you give me an example of one of those?

Yeah, absolutely. And before I begin this, one thing to summarize what I talked about, which is going to be the premise for this answer is I see stakeholders and data analysts as partners with data engineers, and data engineers are not ticket takers. I think that changes the perspective. So data engineers are actually problem solvers and they work sit at the table with the analysts and with the stakeholders. So having said that, there's just a few examples that come to my mind. I'll give you one example where if you look at one of the bigger projects that we undertook over the last year was migrating a data warehouse and our visualization engine. So we were on Redshift and Tableau as a visualization engine. Redshift obviously was our data warehouse. We migrated from there to Snowflake as our data warehouse and Domo as our visualization engine. Now, it was not a straight up migration. What we ended up doing is we had, let's say around hundreds, 400 plus data sources that were used in the old world. We were able to condense it to one tenth of that. And that is the reusability that I would like to encourage. Now, that is still higher than where I would expect it to be, right? So the next phase would be how do we go 1/10, not 1/10, maybe 1/4 of that right number. And that is the reusability that I'm talking about. What that does is you actually have a single unified definition of metrics. The core data sets stay the Same. Now the stakeholder, whether it is somebody from finance or somebody from ops or somebody from marketing, can still access the base core data set which is certified, and they can still slice and dice it in the way they want. But the underlying definitions, underlying tables and the underlying data will not change. Right? And that to me is the most important win coming out of this reusability or this new notion of having partners, data engineers as partners. Because that in the absence of that, what we had in the past, and I've seen it in other places as well, was marketing would have a different definition of a metric, finance would have a different definition and we couldn't agree on the numbers, right? The numbers said different story. Now with a reusability pattern as a, let's call it reusability access pattern, just for this discussion, that allows you to have a single definition and that that also an offshoot of that is we have also built good documentation in the form of data dictionaries, in the form of to improve self service access. That's an example that comes to my mind immediately where reusability is a big win, not just for the business, but for the analysts and for the data engineers as well.

That does sound tremendously important. I think that's something that often isn't obvious from a business point of view, that when analysts are reinventing metrics for every project in a slightly different way, then these lead to like chaos and that you're never quite sure what the right number is. So yeah, that reusability does sound tremendously important. You mentioned that you're just going through migration from I think Redshift and Tableau to Snowflake and Domo. I know that migrating these technology stacks is a complete nightmare for everyone involved. So I'm just curious as to your motivation about what you decide to switch.

It is a nightmare, I would say if you don't plan it well, it is going to be challenging. The big motivation for us was, I would say there's a few of them, the first one I already highlighted, which is we had too many data sources, too many reports in the old world. Now we wanted to get to a world which is a lot cleaner, lot more reusable, lot more flexible because you can have wide data sets and smaller number of data sets. So removing tech debt, I would just classify as that tech debt was one of the biggest motivators. The second was really you don't want to be in the business of maintaining or upgrading or keeping infrastructure up to date. You want to be in the business of running the Business, which is I want to be able to get my stakeholders the data that they need for strategic decision making and be a partner to them and move much quicker than having to maintain and upgrade these servers. Right. That was the second major reason. And that is one of the themes that I've seen emerge in the industry over the last, I would say five to eight years. And it's just growing, which is a great thing, which is the use of managed services. When you have aws, you don't want to be, or an Azure and so on, or GCP for Google, you don't really want to be in the business of maintaining your own servers. So similarly, if you have a cloud based service, why would you maintain your own infrastructure? So DOMO was an example of that. We looked at other vendors as well. They're also cloud based. Domo kind of checked all the boxes for us. Now to answer the second part of your question, which is, yes, it was challenging, but the way we did it is we looked at it from all angles. We knew where the North Star was and decided that even though the North Star has come to the a clean state where the data sets are really small and unique data sets and we have reusability, we knew that we couldn't achieve all the way there because at the same time we are in the middle of this migration. So if you change too many parameters, if you change too many things in this migration, you won't know what broke or what caused something to break. So we were very methodical and we built out a project plan and we were very clear in what are we trying to do here. And the other thing that I would say going back to their motivations was performance. Snowflake is very extensible, it's very flexible, it auto scales and you can add capacity as and when needed, as opposed to the old world of redshift. And we had different kinds of stakeholders and we were able to build different, call it virtual data warehouses and we were able to build different data mods for different stakeholders, whether it is operations, whether it is finance, whether it is marketing. And this is another big motivation for us. So the way we executed on it was, I would say the planning was a big part of it. The second piece is we decided not to migrate them one after the other because what would have happened is if we migrate, just let's say for the sake of argument, if you migrate from tableau to Domo and leave redshift alone, then you are building a bridge from, instead of going from redshift to tableau now it is going from Redshift to Domo, which is Throwaway. Now we have to build another bridge from Snowflake to Domo. So we said, let's do it. At the same time, it was a bold decision, but also we did a good amount of planning around it and we phased it out so we actually out of it, we are out of Redshift, we out of Tableau. We only have Snowflake and Domo. And so now phase two is beginning, which is now that we have taken care of, I would say, 80% of the tech debt. That remaining 20% is what we are attacking now in the form of different projects.

That's absolutely fascinating. I'm sure the planning must have been incredibly extensive, but I found it interesting that you wanted to shift everything at once, which is it must go against the sort of agile instincts that I think most data engineers have, because you do need to, as you say, you would be sort of creating redundant connections. Otherwise. I'd like to talk a bit about who's involved in your data engineering team. What sort of roles do you have within your team?

First up, we actually club the data engineering team and the data analyst team together. So there are three separate roles and I would explain them within this org. And there are very thin lines between them, so they are not hard and fast. Right. And I talked about why we decided to merge those two teams to encourage really good collaboration. So really three kinds of groups, three kinds of roles. In this group, one is data engineers and inside data engineers. There are two kinds of roles. One is more of the ETL engineers who are responsible for cleaning and building those pipelines, whether it is airflow, whether it is something else. Again, they are not clear distinctions. Right. There's a overlap in them is people who are seasoned and who are good in dimensional modeling. And you can see the forest for the trees. That's the way I would put it. Instead of building a solution that works for a given set of use cases, they would look at it holistically and see what are the common access patterns and what are things that go together. What are things that we can put into these core data sets? So that's the data engineering team. So I have data engineers. Some are more modeling focused or dimensional modeling focused, and some are more ETL focused, but they are still data engineers. And then we have data analysts and inside data analysts. There are two subgroups. One is focused on what we call product analysts. Right. They are focused on the core E Comm experience. They would work in this squad model and they would look at what is happening at the top of the funnel, how are members getting in, like the funnel analysis and so on. And also for repeat orders, people who are placing orders, the core ECOM experience card checkout. Then there is the other set, which is, I would call it business data analysts, right, who are now embedded inside these orgs, let's say operations or finance. And they act as a partner to these orgs. For them it is like an extension to their team. And for us they are like extension of our team. So it works very well because of this embedded approach that we have. So as you can imagine, right, there is a big overlap between these three roles. And so the role that the title that has been floating around a bit is analyst, engineer. We don't have anybody with that title. But on a day to day basis I would say most of them are doing similar role because of the way we've set it up and because of this strong collaboration, we have been able to build not just execute quicker, but also build with quality. Now I'll give you an example, right, in that project that I just talked about of the migration, why were we able to achieve both data warehouse and the visualization engine, swapping both at the same time? And we got efficiency out of it. The reason for that was the strong connection and the strong collaboration between the data analysts and the data engineers, right? So we were building validation layers or validation suites at different steps of the process, starting with the data lake at the very bottom to the data marts, of course, data warehouse and the data marts. So between all these three, we were able to rectify and be able to look them side by side, what happens in the old world, what happens in the new world. And if the definitions have changed, can we explain it? And we'll be able to bring the error rate or the divergence to be like less than 0.5% or less than 0.1% in many cases. And that is an example of a strong collaborative team. So I'm of the strong opinion that you have to get the analysts and the engineers working together because that is where you achieve great things in terms of reusability, in terms of a single unified definition, and in terms of building things that are very performant at the bottom, but also very flexible at the top. And what I mean by that is the visualization engine should be simple. You don't want to have 10 page SQL queries because that is hard to debug for you and me. It's even hard to understand what is going on Whereas if you break it up and this is happening because of the strong collaboration between data engineers and data analysts, we've been able to build a lot of heavy domain knowledge and business logic at the bottom layer, the data lake and the data warehouse and the data marts. So the visualization engine is very simple. You just do a select star or select star with some where clauses. That's it. And all the things that are below it are the ones that are highly validated and clean and certified.

That's absolutely fascinating. And I do find it interesting where you've got this interaction between the data engineers and data analysts. So I'm curious as to do you find that they can speak the same language about these data problems or are there any cases where you find they have differing opinions or the terminology is different and they get confused? How do you find they interact with each other?

The skillset, if you look at the skillset of the two groups that I mentioned, they're slightly different and that is there for a reason. But as far as the language is concerned, they're beginning to speak the same language because they've been collaborating so well together. Now, at the end of the day, what this really encourages, I would say is smart problem solving. At the end of the day, what are we trying to achieve from this? We want the data to be reliable, we want the data to be available to have quality and trust in the data. And ideally we want the data to be self serviced. Right. I really don't want the data team to become the bottleneck. So what this encourages is the data engineering team, because of the strong collaboration now, really understands how is the analyst thinking about it from the business standpoint and the analyst understands what are the challenges that the data engineers need. So because of this, to answer your question more directly, yes, they are speaking a language that is a mix of the two, which is great because then both sides see things from the other's perspective and that really allows us to build solutions that are more testable and that are more reliable and really extensible. And to add another point on this, right, in a standard E commerce, right, we are an E commerce company so we shouldn't have to reinvent the wheel every year. You shouldn't have to create new metrics on the fly. The number of metrics that we, the core metrics that we use should be like, it's well understood. Now the question that comes up is how are the stakeholders and analysts are really representing the stakeholders going to be looking at it? How are they slicing and dicing it, that discussion wouldn't happen organically, but because of this setup that we have, it happens organically where we solve problems together. And so there's an architecture design. Both of them review it from different perspectives. And so in that way we have been able to solve, I would say we've been able to move faster and also solve problems which are. I mean, we have been able to solve problems before they appear as problems. Right. Because we know we can actually see. Again, going back to that analogy I was giving earlier, one of the big things in data engineering is you should be able to see the forest for the trees. You shouldn't just focus on the problem at hand, but look at the common access patterns. And this setup really allows us to do that. And of course, in terms of lingo, yeah, they're speaking very similar. That is where that analyst engineer is a very interesting thing, a phenomenon that I see in the industry. It is happening.

Do you want to talk a bit more about that? Like where the analyst engineer would sort of fit in within your team, sort of, if they exist, or within a sort of broader data engineering scope within an organization?

Yeah. So I have a few analysts in my team who have expressed interest in transitioning to be an analyst engineer. And I love that. That's a great outcome. So I definitely see that as a good transition point. And the way I would position it is somebody who is still an analyst, their skill set is still an analyst. But I mean, as an analyst, you are strong in statistics, you are strong in understanding the business stakeholders, great communication ability. And the same time you want to be more into the nitty gritties of how the, let's say, dimensional model is built or how the data sources are organized or the etl. I see it as a transition. I don't see it as a final state. Similarly, on the data engineering side, if you want to be. This setup really allows you to see what is the end goal of my analysis or what is the end goal of this data set that I've created. And there can be a case that I can see where a data engineer would want to do more analysis on top of maybe 80% data engineering and 20% analysis. And as I said, it's a slider. It is not really a thick line between the two. Nobody has reached out yet, but I can see that happening too on the data engineering side. But data analysts, for sure, I've seen a couple of them have expressed interest in moving to an analyst engineer.

Excellent. And so I'd like to talk a bit about what's going on in retail. And I'm curious as to whether retail data is different. If there are any quirks of the industry that have an impact on what you're doing both in data engineering and in data analytics.

Yep. Now, having spent time in tech companies which are also direct to consumer like Netflix and Facebook for example, the data needs and the data sets are pretty similar. The problems that you're trying to solve are pretty similar. The problems are around scale, around availability, around data quality and so on. In the retail space, especially in the space that we are in E commerce, Amazon is in the same space, for example Instacart and so on, you would see that the problems are similar to what you would do in, let's say an Instagram or a Facebook. But the data, but the actual metrics are different, the stakeholders are different. To give you an example, the stakeholders that you would work with in retail would be merchandising, for example, in our case as well, which is people who actually talk to the vendors and bring those products onto our site. You would have the marketing team which is responsible for spreading the knowledge about thrive market and the two kinds. One is acquisition, acquiring new customers and one is retention, retention campaigns or outreach or CRM. And then you would have operations, which is again not something that you would see in a tech company. But in retail you would see it where, because your life doesn't end when somebody does some interaction on the website, you also have to ship that order, physical order to them, similar to an Amazon. Like merchandising, for example, and operations, these are different for retail. And of course there are other, like finance is finance, Finance is everywhere. Right. So then there's member services as well, like people who pick up the phone or chat. There are things that we can do there as well. Now, having looked at it from that angle, if you look at the actual metrics and the data sets and the what makes retail different, the first thing I would say is it's the metrics and the fact that you are looking at it holistically. As I said, it is not just an app that you play with. It's not a app like an Instagram where you upload photos and you forget it. Here you're making a full transaction. So the most important thing is what happens from the time you onboard to me providing you an experience where you can browse the site, you can search for things, we can recommend things to you, we can personalize it right off the bat. And we can do that much better than if you're walking into a brick and mortar, because we have so much data about what you did, and we can use that really amazing experience for you. And then cart and checkout, you have to be able to put things in your cart. You have to be able to pay through your credit card or Apple pay or whatever, and then those products have to be shipped to you. The entire cycle is different in retail, and you have to have metrics that attack every single part of it. Order to cash is obviously one of the most important problems. And order to cash is not just you have to be able to account for every single penny, but also the tricky part. Here is the metrics again, I was going back to some of the metrics. The metrics could be based on shipment date or order date, which are different. I could place an order now on the site or on the mobile app, but the order may actually be shipped maybe, let's say in the night. And the order may arrive maybe in a day or two days. Right. So depending on what you're looking at, each vertical on each domain will have its own set of metrics. So merchandising is more concerned about providing visibility to the vendor experience for all the vendors, like how is their performance because they're paying us for these products and so on. Marketing is concerned with every dollar that they're spending, whether it is acquisition or whether it is CRM. Can we track it back to how the customers are reacting? Similarly, you know, operations will be focused on the time that the order was placed to when the order arrived at your home and all the sips that happen in the middle. So you have different state transition diagrams for different teams. And that is different. And that is so exciting about retail and e commerce. And I'll just say I talked about this example, brick and mortar. One of the big things with e commerce is because you have access to what the users are doing, you can actually customize and personalize their experience right off the bat and also in every single session. And that is something that, to me, is very exciting. Retail, especially e commerce, has a nice mix of an app kind of setup where you, let's say an Instagram or a Facebook or a Twitter, and then you have the brick and mortar experience of shipping things. And so you can actually cover the entire gamut of the experience.

So it sounds like there's quite a complex setup because there are so many different stages to an order and you've got different places you can order and things like that. How do you manage keeping everything in sync, having all that different complexity in a Form that is sort of consistent and understandable.

This is where the data engineering plays a huge role because this data is collected at different places. You can have data coming from the website around user interactions. You could have data collected from the shipping. How are the shipping vendors, for example. All this has to be ETA'd. How? All this has to first land in a, in a data lake, let's call it in its raw form. And then we have to transform them, do multiple transforms to make it available in a data warehouse. From there we have to be able to now surface reports, let's say in Domo, in the form of dashboards. And of course, a flip side of it is you also want to see alerts and monitoring and smarter alerts, ideally. One of the things that we have been able to do is if you look at the, the entire pipeline, we have been able to build validation layers at each step, validation layers at the data lake, at Data Warehouse and Data Mart. And then we've also set up alerts. And these alerts are based on some checks, data quality checks that we have put in place. If, let's say a column is empty, a column that is required, or we see a deviation which is more than two and a half standard deviations from the last two weeks or last month, we can capture some of those vagaries and erase it and flag it. And the earlier we flag it, the better it is. You don't want to capture things at the very last layer, you want to capture it at the data lake layer or the data warehouse or the data mart. This actually allows us to be very thorough in the way that the data is landing and also the way the data is used.

I like the idea of capturing the problems earlier. The worst case is like it gets caught by the customer after you've made a mistake. So, yeah, I think solving the problems as early as possible seems like a good idea. And you mentioned that you have these tools for just validating the quality of the data at different stages. I'm wondering, have you made use of artificial intelligence tools at all in order to help you with this?

We haven't used AI per se in this process yet, but we are getting there. So we actually have plans in our roadmap to start tackling them and start using them. So some of the tooling, some of the tools that we have used are of course, in Snowflake you can set up all kinds of alerts. In Domo, you can set up all kinds of alerts. So we have Python utilities and we use airflow for Scheduling a bunch of these jobs. So we do set up alerts which then automatically they go to Ops gd, which is similar to Patient Duty. Right. A vendor that we use and also Lancep is an error email or a Slack in one of the Slack groups. And these Slack groups are also open not to data engineers alone, but also to stakeholders when the issue is actually downstream. As a result of these tools, we are getting smarter. Now in the future, as I said in the second part of the year, we are going to be looking at some data, automated data quality checks. We are looking at vendors. We use AI in a big way on data personalization on a different layer like to optimize experience and so on. On the data engineering side, we haven't used it yet, but that is definitely the direction we are going in.

And you mentioned you've been using like Pager Duty and some of the tools. Can you tell me how you've been using these tools, how they fit into a bigger sort of data governance program going.

Let's start with the data checks as an example. If the let's say for a given day you see that the marketing spend went up to 100,000 in a certain single campaign and the limit is 1,000, making it as an example, then you know that something went wrong. So it automatically triggers an alert in obsgeni, one of the pipeline jobs will trigger an alert in obscgenie which will then be acknowledged and handled by somebody. Now that is part of our Obscini and Airflow is. And Slack of course is part of our stack. We also have native alerts in Snowflake and Domo that also allows you to capture some of these depending on the different layers. We also have integrated these things as part of CI cd. We use Concords as a CICD platform. So as these jobs are even at the time of deployment, we run some checks. And if I look at the rest of the stack, if you look at what this setup allows you to do is it allows you to be to not just capture things before the customers have got it, but also be more agile and also look at what look for problems. So going back to that question on how we would want to use AI, one of the things that I would like to see is if you look at the common errors that we have seen over the last, let's say 30 days, 60 days, what kind of patterns can we derive from them? What are things that we can proactively use? Right now let's say we use a two and a half standard deviation. Should we get Smarter. So those are things that will definitely come into play. And the other theme that I'm looking at are things like more abstraction for the data pipelines. So there are tools like DBT and others that are floating around that are becoming pretty standard. We will be looking at that. And we're also looking at tools like Soda for example, which will allow you to work with the stakeholders when setting up alerts. And again, going back to my model that I talked about of data engineers being a partner to the stakeholders, using tools like those will allow the stakeholders to also play a role in data quality. And so going back to your question on data governance and data quality, that is definitely one of the biggest, big, big foundations for strong data engineering and we are definitely focusing on them. And one of the things that they do for you or the goal, I would say the North Star for me is as the needs of the or the data needs of the organization increases, we are using data everywhere in thrive. But as the analytical needs of the business increases, you don't really want to be growing the data engineering and data analyst team. What you want to be doing is building tools and governance and self service tools that can then be exposed to these stakeholders, the end users, so they can be assured that the queries that they're writing or the data that they're pulling or the metrics that they are slicing and dicing is actually accurate. And so that is where data governance will play a huge role. So I would say data governance, yes, it's important, but the end goal and the North Star really is driving self service access and really scaling up the teams. At that point, if you don't want to be growing the team with the size of the business, you really want to be growing the tool set and the data dictionaries and improving data governance and certification.

That's interesting. So you're suggesting that the way to scale your data engineering is really just about making self service access as great as possible. Do you have any other tips for this on how to how you can improve self service access?

So self service comes with, I would say a few things that are pretty much the prerequisites for self service. The first one is data has to be available and reliable and something that you trust and that is for data quality is like a given. If you don't have data quality you cannot go to self service. The second thing would be data dictionaries and good documentation on and thinking as a stakeholder, which is where working in a squad model helps you. So you want to be thinking of okay, if I'm in marketing and I'm looking to look at sales in a given cohort, how will I do that? Using the existing tables. Right. And really increasing our democratization of the data, as well as knowledge of the schemas that are used and the tables that are used. So that to me is the second layer. So the data quality is the first layer. The second layer is the data dictionaries or data documentation. The third layer would be what we just talked about, data governance. Now, as a stakeholder, I could be building some metrics, but I want to be assured that what I'm doing does not overwrite the main definitions or I'm not doing anything that can actually compromise the quality of that metric. And so that is where governance comes in. And the fourth layer, I would say after that would be. As you do this, you should also look at cost optimization. Like if you have unlimited resources, yes, you wouldn't worry about cost. But in this case, efficiency and optimization plays a huge role. So as a data engineer coming back from the data engineering lens, one of the most important things that I would say I would put even as lower than these three ladders or the three steps of the ladder that I talked about is efficiency or optimization as a goal. If, let's say as an example, if you are taking, if you're running a daily pipeline to generate the revenue for the last day and it takes you six hours, what can I do to bring it down to maybe 20 minutes? Should I run it more frequently? That's a perspective that data engineer needs to bring to the table as well. Because it's not just cost as a dollar amount, but also cost to the business. The sooner the data is available, the better it is and also the more reliable the data is. Those are the four angles or four pieces of advice that I would give you if you were to really scale up data engineering. Org.

That's interesting that you said going from having data that's sort of 60 minutes out of date to 20 minutes out of date can bring you a benefit. And it sounds like is the end goal there, you're hoping for sort of real time data everywhere or how close do you need to get to that?

Yes, that's a great question. It depends on the use case. We do have some real time data sets that we use today and in some cases we are fine with an hour of delay, we are fine with 15 minutes. In some cases we are fine with even a day. It really depends on the data set and the use cases. I would echo the same point if you get it closer to real time, it is not just efficient for the stakeholder to make business decisions in most cases, but also it's more efficient for the data engineering team and the stack because you may have to just deal with the deltas, incremental loads. And if you lower the frequency or if you, sorry, raise the frequency and lower the interval, you're dealing with much smaller data sets which you can actually be more performant in there. And so I think going back to that other theme, efficiency is one of the main lens that a data engineer should have in their repertoire. Like it's one of the main things that they should be focused on. How do I get efficient? And efficiency is, as I said, different can have different pillars. One of them is cost, like the dollar cost. And there are others as well, which is the time to make this data available. And even the quality of the data is something that I would put as efficiency because you don't want the team to be fighting fires. And this is where AI will also help. Right. And that's my goal in the near future. I would like to see how can I, once we know that there is a problem, instead of having a human come in and rerun the pipeline. We do have retries and all that, but sometimes retries is not going to solve it. And this is where AI comes in. How can I have smarter retries, like fix the problem and then rerun it, which normally requires a human. And this is something that is not a dream. It's available, it's something that we can do. And that is one of the things that we'll be going in is how do we self correct? How do we have the system self correct and rerun so there is no human involvement.

That does seem like the dream is okay, something's gone wrong with my data engineering. Let's have the system fix itself and try again and hopefully get it right this time. So you mentioned this idea of sort of efficiency and automation a few times. I'm wondering if there are any areas of data that are easier to make more efficient or easier to automate.

I think everything is easy to automate. And I wouldn't draw a strong. Like it really depends on. It depends on the use case. I would say I would start with the end goal in mind. If we are automating it, what does it solve? And again, this is where you have to have a holistic point of view. And this is where having the analysts and the stakeholders working together will give you that perspective. I can give you an example like on the ops side, we built operation side, we built some dashboards. But we don't just go ahead and build them, we work with them to understand what are the latency requirements, what are, what are the different views that will be needed and what are the different dimensions or what are the different ways you will slice and dice it. So that to me is really the crux of it is understanding and doing it holistically so you don't have to build something that only works in a given use case. And so that is where I would say when you talk about automation of the data set, it really boils down to the use cases that are driven from that data set. Any data set can be automated. And I'm actually there have been cases, for example in marketing where we had manual uploads when we had manually in the past somebody entering the spend for the last day on marketing again as opposed to that. Now we are pulling it through APIs and all the. Let's say I'm using this as an example. But you can do the same thing for in any case, whether it is merchandising, whether it is marketing or whether it is some other. There are APIs available. So when you talk about automation, the first thing you should do is see if there is an API available. So there is no human in the loop. So there are no choices. You can always have fat fingering and things like that. If you have an API in the loop, then that takes care of that problem. And then you have these. Most importantly, the data ingestion is the first step. The second step is data quality. Now you should have checks on is my raw data accurate? And as I said, the more downstream you cache the problem more, the better it is for everyone involved. If let's say that checks off, then you can look at the next layer. Now how am I using this data set or this data and what are those derived tables have checks on that level as well. And so that to me is the way to again think holistically from the use cases, but also drill down into each of the layers and automate all the way.

So it's just layer upon layer of checks all the way through just to make sure you've got data quality at every stage.

Yep. Okay.

Since there are a lot of people who are curious about getting jobs as data engineers, I'd like to know what sort of the skills you look for when you are hiring.

Really, if I'm hiring for data engineers, there are two kinds of skills that I look for, right? Or maybe three kinds I would say one are hard skills, one are soft skills and one is how seasoned, how experienced you are. Now let's start with the hard skills, right? If you look at data engineering, I talked about some of the stacks. A high level language like Python is very popular. SQL and Python are going to be your bread and butter. You have to do that every day. Also knowledge of dimensional modeling and data modeling helps because you can now see the forest and look at the common access patterns and experience the data warehouse. That's the hard skills part. Soft skills. I need a very strong ownership mentality. Somebody who really thinks of it as their own business, thinks of the workplace as their own business. And see how can I think like a stakeholder? How can I look at it, look at the goals behind this data pool or this data set or this report as opposed to just doing what was asked. And that is where communication, two way communication is very important and strong communication ability and strong collaboration ability is important. The third area would be strong seasoning in one of those, let's say tech companies or e commerce companies and stuff like that. When I was in Netflix we were not looking for people fresh from school. We are doing the same thing here. We really need people who have done a stint in some other companies, right? 5 years, 8 years, 10 years depending on the level of experience. Because we are a very fast paced startup going back to thrive market and we really want ramping up is going to be quick for us. Now we have a good process in place but we really want people who are experienced and have done that before. That's my advice to people who want to get into data engineering. The third section around seasoning is really what I'm talking from putting my hiring manager hat on in thrive market. So those are three things I look for. But for people getting into data engineering, the hard skills and the soft skills are things that I would recommend. You need both of them.

Yeah, and so I mean that does sound like the seasoning thing is going to be the hard part for people who want to break into the industry because you can't have experience until you've got experience. It's always a bit of a catch 22 situation. But you mentioned ownership. Ownership has been one of the important sort of soft skills.

Maybe.

Can you give me an example of how ownership works within your data engineering team?

Going back to the example, let's say a metric is broken, let's say a metric is wrong. Revenue. I would like to understand what are things that I can do so it doesn't break again. And looking at it, the ownership mentality comes in, in that you think of it as. It is important for you as a data engineer to understand the business value of that metric and the business value of the pipeline being up to date and the data being up to date and reliable. To me really, ownership mentality means somebody who treats this company as their own and really don't look at it as, let's say a 9 to 5 or 9 to 6. They look at it as how can I really make impact to the end user? And the end user in this case could be an analyst or it could be a member, like actually a member on Thrive Market, or it could be a stakeholder. So I think that to me is. And if you have that mentality, then you will not look at solving problems in a silo. You will not look at solving problems in a limited fashion. You will look at it holistically. And again, I've used this term a few times. Common access patterns or the common scenarios or the common problems that I've seen. Like what are other things that I've seen? I'm not, okay, I'm looking at this particular problem, but what are other things that I've seen in the recent past that could be related to this? So can I look at it holistically and solve all of them together or a big chunk of them together? That to me is ownership mentality is going above and beyond and really thinking and asking a lot of questions. You may not know all the answers, which is fine, but having that mentality will allow you to really question, like even for a metric that you are developing, does that exist somewhere else? What is the end goal that the stakeholder has? What are they trying to achieve? That to me is ownership mentality going above and beyond in terms of the scope, the requirements and the solution.

Nice. So it's really about being able to sort of think about problems even before you've been asked to try and solve them. Just think about what's going on in data engineering right now. Is there anything that you're particularly excited about?

Yes, I'm very excited that we have finished this migration. Talked about of course a challenging project, but we did a good job, an excellent job at that. Now the next thing again, remember my North Star is always self service access because that is the only way you can scale up. Now in there we talked about those three pillars, data governance, data quality and data documentation. We are making good progress in all of them. And to me, if I were to pick one, I would pick that self service as the big things on my mind and the big things on my data team's mind. If I were to pick a second one, it would have to be consolidation or really looking at consolidation of the data sets. And remember we talked about going down to one tenth of the number of data sets in this migration. Now how do I go from there to 1/4 of that and really have like data sets that can have better self service? And that is where data dictionaries is also going to be a big chunk of our work. Very excited about that. And I'm working and the team is working closely with the stakeholders and the feedback has been very positive. And depending on the level of familiarity of the stakeholders, some people may want to do self service at the domo layer, at the visualization layer, some may want to do it one layer lower who are familiar with SQL, they want to look at the snowflake layer. Right. So that to me is very exciting because A, there is certification, there's data governance in place, B there is data quality and C, there is data dictionaries and tooling in place. So once you have these three, self service really becomes more than a dream, it becomes a reality.

That sounds great. And I do like the idea that you need to provide access at a self service level for people who don't.

Want to write any code, they just.

Want to point and click or just stare at a dashboard. And you also need that other layer where people might want to sort of drill in and write their own code queries and explore the data themselves. All right, so do you have any final advice for organizations wanting to improve their data engineering?

So if I were to summarize it into like three points, I would say the first important, most important thing is understand your stakeholders. And when I say understand, see the common patterns, see the what is their North Star, what are they really trying to achieve so you can look at the problem holistically. The second big thing is self service. That's the only way to scale up and that will you really want to be in a place where you could be chilling at the beach in Florida and the stakeholders have self service access to that data. That's how your world should be. And even if there are issues that are raised, issues that you caught through your alerts and monitoring, they can be self corrected by the engine itself. Now we have some ways to get there, but that's the goal, right? Have a very clean way for clean and easy and accessible way for stakeholders to access their data and trust their data. And this is where we talked about data governance, common utilities, documentation and so on. So that's the second one. Drive self service. And the third advice I would say is embrace managed services. Just like we did. You don't want and we have done it, not just in the data team, but even outside of it. You have tools out there and we are in a very fortunate time right now where you don't have to reinvent the wheel, you don't have to start from scratch. Your goal should be how can I add maximum business value and how do I get myself out of this business of maintaining the infrastructure. So those are the three pieces of advice I share, I'll give to data engineers, which is again as a reputation. Understand your stakeholders and the requirements. Build self service or build towards self service. And the third is embrace managed services. That's your friend. All right.

All sounds like great advice to me and I really do love the idea of chilling on the beach and all the data services still working. That sounds like a dream. Okay, thank you very much for your time.

Love. Thank you. Richie was great talking to you.