#166 Optimizing Cloud Data Warehouses with Salim Syed, VP, Head of Engineering at Capital One Software

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

When it comes to cost optimization, it's never a once and done deal. You have to be vigilant. It's a constant battle, so you need to be vigilant. You need to be watching for new users, new usage, new usage pattern. Workloads are always introduced in the system, so don't focus on reducing costs. Foreign.

Welcome to Data Framed I'm Adele, Data evangelist and educator at datacamp, and if you're new here, Data Framed is a weekly podcast in which we explore how individuals and organizations can succeed with data and AI. I think it's safe to say that effective data management is essential for any organization looking to succeed with data, and most of the time organizations look to storing their data in the cloud to begin that journey of data management. However, things can get easily out of control as you're transitioning to the cloud. Costs can balloon as you set up your infrastructure, data governance becomes more and more challenging, and a lot more so. How do you make sure you navigate that cloud journey effectively? Enter Salim Syed Salim is the VP and Head of Engineering for Capital One Slingshot, which is a tool Capital One software developed to enable organizations to optimize their data in the cloud. He led Capital One's data warehouse migration to AWS and is a specialist in deploying Snowflake to a large enterprise. Salim's expertise lies in developing big data lakes and data warehouse strategies on the public cloud. He leads an organization of more than 100 data engineers support engineers, database administrators and full stack developers in driving enterprise data lake, data warehouse data management and visualization platform services. Throughout the episode we explore cloud data management and the evolution of Capital One Slingshot into a major multi tenant SaaS platform. The shift from on premise to cloud based data governance the role of centralized tooling as you transition to the cloud Strategies for effective cloud data management including data governance, cost optimization and waste reduction, as well as insights into navigating the complexities of data infrastructure security and scalability in this data driven age. If you enjoyed this episode, make sure to subscribe to the show, give it a rating, share it on social. We'd love to hear your feedback. And now onto today's episode.

Salim Sayyid, it's great to have you on the show.

Yeah, happy to be here.

So you are the VP and head of Slingshot Engineering at Capital One Software. So maybe first to set the stage, walk us through Capital One Software and Slingshot and your role as its head of engineering.

So Capital One Software is an enterprise B2B software business of Capital One. It's dedicated to providing data management solutions to companies that are operating in the cloud. And our foray into software business, it started a long time ago with building so many different softwares in house softwares. At the time we were in the cloud, we just didn't have those capabilities. How do you manage cost? How do you manage efficiency? How do you do data governance in the cloud? All these we had to build ourselves. And then we also realized that we're not unique to this challenge. There are other companies that are facing similar challenges while they're going to the cloud or already in the cloud and now experiencing these challenges. So our offering is to help other companies be more efficient and in the cloud.

That's great. And maybe a bit expand a bit on Slingshot as well and what Slingshot is.

So Slingshot is our first B2B software and it's designed to help. It's a data management solution that helps companies optimize cost and accelerate their journey into Snowflake as well as it really allows you to remove your wastage and inefficiencies from the way you're using your data platform.

And I want to center today's discussion on really how organizations can maximize the value of managing their data in the cloud. You know, in many ways, centralizing your data in a cloud data warehouse and building a modern data platform is really table stakes for any organization today trying to be data driven. A lot of data leaders have this top of mind, so maybe kind of getting a bit of background as well. Walk us through why that is and why nailing data management and data infrastructure should be top of mind for every data leader today.

The way you get value in a business, it really starts from data, right? It's understanding data. The insights, the insights that are coming out of data is going to drive your business. It's going to help you with your customers. It's going to help making their lives easier. So data is very crucial. And what we're seeing Is that when you move to the cloud, the amount of data, the type of data, all this increases exponentially. Right? If you have all that data coming in and it's not well managed and well governed, you don't know where the data is and the right accesses around it, right? Security, all that will make it more challenging for you. Either you will not have business, cannot get insights in a timely manner, or you will create silos that will prevent you from accessing each other's data and seeing the value that comes with sharing of data across your organization. So to have a good data management is really crucial from the start.

I couldn't agree more. Because ultimately you're creating that pipeline right from collection to enable people to draw insights. And that pipeline needs to be resilient and needs to be highly efficient, well managed pipeline. So maybe zooming out on the broader context of the industry, you work with a lot of industry partners and have worked on data management extensively at Capital One internally. Where do you think we are today when it comes to organizations effectively managing their data? Where are we on this journey as organizations mature their data management?

This is a very good question. Different companies are different places in their maturity of data governance and adoption of cloud. But what I'm seeing is that one of the places where I see some friction is that a lot of companies are moving to the cloud, but they still have the mindset of on premise. And let me explain what that means. That means that in the on premise world, you had a centralized team managing all of your data, right? They're the ones who are saying, hey, this is the data you can have. They're doing the data governance, data publishing, all of that. And that really is hard to maintain in the cloud, especially because now you're going to have an explosion of data, as I explained. And this is all the businesses really wants to go at the speed of their demand and they cannot be dependent on a central team. And the central team doesn't have the domain expertise across the lines of business, so they become a bottleneck in the cloud. So that's where we see a lot of friction around adopting this new mindset where you want to allow the businesses to move at the speed that they need. But also there's two things you have to really be careful of. One is creating silos and the second is how do you enforce enterprise centralized policies, governance policies across, uniformly across your organization, and that to find that balance Capital One has built. The way we look at things are you have central policy and central tooling, but then federated ownership. What that allows you to do, centralized team to do is create the policies, make sure the tool enforces them or puts the right guidelines guardrails around how you publish data, how you access data, how you govern data, but then gives the ownership for the line of business so they can move, they can publish on their own, they can find data on their own without having a bottleneck on the central team. And I see that as the biggest challenge for companies to really truly adopt the cloud mindset.

Okay, that's really great. And maybe kind of expanding on that cloud mindset. What have you seen or ways to deal with this particular challenge that you mentioned here on having this federated approach while still centralizing data governance and data controls. Maybe walk us through this kind of solution landscape in a bit more detail on how data leaders should be approaching this aspect or this challenge of data management.

It starts with what Capital One believes in how to solve this problem, which is centralized tooling and centralized policy. What we've built is we've built centralized tooling for our imagine our data publishing. I'll give you two examples of publishing to your data environment and then what consumption needs are. First, let's talk about publishing. So one way to publishing is you create centralized policies, you give it to the line of businesses, they publish data into their data environment. It's very becomes from a central team, it becomes very difficult to enforce all those to guarantee that everyone's following the practices. So what we've done is we've built central tooling that allows you to publish your data and making sure that the publishing, all the governance need for publishing data, for example, registering your data, are you checking for data quality? Are you identifying what is sensitive, what is not? Are you making sure that the sensitive columns like credit card or personal informations are encrypted or tokenized? All that is governance need for publishing, a pipeline. And what we've done is we made that transparent to the business so that if they're using our central tooling, all that comes free. So not only is this going to help you be more compliant, but also by making sure that everything is registered, the data quality is collected, it makes it on the consumption side for our users and employees when they're accessing data, they have, first of all they can find data very quickly, find relevant data very quickly. Because now we have all the data registered in the right places, we're collecting the right metadata and also we provide the consumers of the data enough information about the data that they can trust it. So think about what is the result of your data quality, what are the sample data, min max data profile, all that is given to you so that you can trust it. And then in the same experience, we allow you to access the data, request access to data so that it goes to the right owner of the data and they can approve and you provide your business justification. The point I'm trying to make is you want to make sure that all the data governance needs are as easy as possible, as transparent as possible, so they don't become a hindrance to the developers or the data producers or the consumers to getting their job done right. And you get both benefits. You get the benefit of compliance and meeting all the regulatory requirements as well as it's easy to find the data, it's easy to get to the data quickly and get the value out of the data.

That's really great. And you mentioned a lot here, the data governance aspect to it. But one thing you mentioned earlier was the need to develop this federated mindset and to let business teams also lead with their data while having kind of centralized policies. Maybe what have you seen are effective ways teams can organize themselves around this model as data management as well evolves? Maybe walk us through the different ways successful data teams have been able to organize themselves around, and successful data producers and consumers have been organized as a consequence of this shift in mindset.

The best people who know the data are the businesses. They have all the domain expertise, they understand what the data means, how to use it, and you want them to own it. Right? And that's where the mindset comes, that they're the ones who know it. Give them the tooling that allows them to publish data quickly, allows them to ingest the metadata about the data so that everyone can get benefit. And the other thing you need to make sure is that data has to be with proper access control. It needs to be. You don't want the architecture to limit data sharing between organizations. If architecture allows that, then you can build access controls on top of it. You can build that. But as soon as you create an architecture where you have to copy data out of one place into another and go through multiple hoops, it becomes very hard and it just slows down the data sharing. So data sharing is going to be a very key component of any organization and you want to make that easy. But with the right controls in place.

And we're talking about controls, we're talking about data governance, right. Like this often makes me. And there's a bit of a, maybe a friction sometimes, as you mentioned, between having centralized Data governance policies, but also a federated model. Maybe who should be the stakeholders involved in setting up a data governance strategy? Maybe walk us through in a bit more detail what a successful data governance journey looks like here for organizations as they set that up.

You have to have buy in from your leadership. It requires buy in from the businesses. Businesses have to also create their own governance, own risk officers that allow them to manage their own risk. Because every line of business has different risk around it. Right. You want to make sure that they have the right ownership, they have the right risk reviewers within their line of businesses as well as a central policy. And the way it really works is if you have a leadership that says this is the way we're going to do it, but this is the benefit whole organization is going to get it, then you have a lot of buy in. But without the buy in from leadership, it becomes very hard to start this from the grounds up. So, so that I think is a requirement for a successful journey.

Yeah, I couldn't agree more here, especially on the importance of buy in for leadership. Now speaking of buy in, I think in a lot of ways the other side of cloud transformation and effectively managing data in the cloud is actually ensuring that people have the tooling skills, the ability to adopt data. Right. And which is, this is much more of a people and tooling problem than it is only a data management problem. So maybe walk us through good examples of data adoption that you've encountered at Capital One and Capital One software customers.

It comes really down to, again, I said the leadership, but it's a data mentality. Right. You want to have everyone be educated on making data driven decisions and it needs to show from top down everywhere, even during development processes, you're making decisions based on data. Right. So that's the culture you want to create first. But on top of that you want to make sure that it's very easy to find, access and trust data. So the democratization of data is very important in any organization to allow just data engineers to get access to data, but make it so easy that anyone with the right access and need can get value out of the data. So that, and then the third piece would be just education. Train the trainers and make every data, even during experience where there's inefficiencies in the way you're accessing data. Make it a teachable, so teachable moments. Create more, more. It's a, it's got a trickle down effect across the organization. So that's how I see it.

Okay, that's great. And maybe expanding Here on the aspect of providing access to data you mentioned here, ineffective ways of surfacing data. I think one common anti pattern we see in organizations today is that there's really limited context on how this data is useful and could be used. And a lot of organizations are trying to solve this with a metadata platform. Right. Or a data catalog. I would love if you can comment here on the importance of surfacing metadata and providing that data catalog for the wider organization and what are effective ways that you've seen as well to provide that context for the wider work.

Yeah, metadata is everything and catalog is the key on that. Metadata allows you to know about your data. It allows you to know not just the business context or the technical context of the data, but. But it's actually becoming bigger than how metadata was used. In the past it was about catalog, you kept the business and technical metadata. But now there is a concept called passive metadata, which is all the cost associated security, all the resiliency associated with the data set. How do you track that along with the metadata? Along with the data? So think about a table you have in your environment. How often is it used? Is anyone using it? How, how often is it updated, how much does it cost to maintain that table and how much does it cost to access that table? Those are all passive metadata. And the, the bigger, the bigger view you have across all the metadata gives a company a lot of edge around where, not only about providing information or what data you have in your environment, but also what's valuable, what is used more, what two data sets are relatable, how often people join the two data together to create the insights. All that becomes very important. So you always want to start with a catalog, but it needs to expand on just static metadata. It needs to be alive. Everything that's happening to that data needs to be collected at the same time to give you even more insight into your operational excellence. Right?

Yeah.

This often makes me think about really the importance of providing as well the business context on how data is being used, like effectively, which teams are using it, which queries, which tables is this data used? And yeah, I couldn't agree more here on the importance of seeing how that data is being transformed. Now, as we're discussing about a lot of the challenges organizations face when it comes to data management, I'd love to also learn more about Capital One Slingshot. Right, so maybe walk us in a bit more detail what Capital One Slingshot aims to solve. In a lot of ways, these challenges that we're discussing here today were challenges faced by Capital One. So I'd love to learn how Capital One Slingshot aims to solve a lot of these challenges.

Before I get into Capital One, the Slingshot, I want to explain the concept of optimization. One of the things that happens in the cloud is that when you are on premise, your data platform costs are pretty much static, right? You buy a certain number of servers and and then you have some constraint on how much you can use, but the cost doesn't fluctuate until you buy a bigger server or expand. And it took four to five months to do that. But when you move to the cloud for the first time, cost is now dependent on how you use it and how much you use. So the more compute you use, the more queries you run, the more your cost is going to be. And what that means is there's a room for a lot of inefficiencies that you probably had in the old environment, but it didn't affect your cost, it affected your performance, but it didn't affect your cost. But now it does affect your cost. So what Slingshot does is it tries to help you optimize your cost. But the way it does is in a few steps, right? First is it provides you the visibility into your cost. Where are the costs spiking, where are the cost, high, low, all that. So second, it gives you near real time alerts and insights into the cost drivers. The last thing you want to do is wait until a month end build comes and realize that you had one query running for 30 days, a runaway query. So you want to know about that as quickly as possible. Then the last one is not everyone is going to be an expert in Snowflake to know how to optimize their queries or optimize their server configuration. So what we try to do is we try to recommend you when we give you, we find inefficiencies, we give you a recommendation on. This is why we're seeing the inefficiency and this is exactly what you can do. Or give you a couple of options and then in the tool allow you to make the changes to your settings, right? And all three together allows you to not only save costs, remove inefficiencies, but also get everything that happens in the Snowflake environment is basically all well managed and well governed because we also have a way for you to provision resources, change resources, all through approval process and the proper tagging. So it's well managed, well governed as well. That's in a nutshell, what Slingshot does.

That's great. And when you mentioned Here on Console Optimization, maybe we should have covered that a bit more depth. Maybe walk us through the drivers behind ballooning costs when trying when managing data in the cloud.

Yeah. The way I see it is there's four different areas where cost can be so variable and have an explosion if you don't manage it well. First one is the compute cost, which is in the modern data warehouses. As data platforms, you always have a compute aspect which is separate from your storage and the more queries you run, the more you spend. The second one is just the query itself, the query optimization. Third one is your data set optimization, which is how much data do you store? How is it being modeled correctly? And the last one is environment optimization, which is around there's going to be a lower environment dev, QA and production. And in these there's a lot of inefficiencies that can happen in the lower environments as well. But let's go back to the first one, the compute. This is where what we understood is that there's no one size that fits for all times of the day. What we see is that workload usually goes up and down based on the time of the day, day of the week, and you want your snowflake resources to size up and down accordingly. And today Snowflake doesn't let you do that. It gives you one size and you have to. So it's very important to know the ups and downs and making sure your compute goes with it. That way you're spending the most effectively. Also making sure that your servers are shut down when not in use and you have some timeouts in your queries so that it'll run away query keeping a server up. So watching for those is very important. The second one is query optimization. Like I said, queries, the longer the query runs, the more it costs. So if you have a badly written query like does, for example a Cartesian data join that can run forever, all that will cost you a lot of money and you won't even know about it unless somebody's watching that. So it's very important to not only built in alerts or runaway queries or but also provide a way to give you advice on how to rewrite the query and get the best performance and lower the cost. And our Slingshot tool does that also. And the third one is data set optimization. I call it data set optimization because it's got a few aspect to it. One is storage cost, even though you'd think storage cost is much cheaper in the cloud compared to on premise. But when you're dealing with multi petabyte of data, then Rich Card will sneak up on you. So it's very important to have a retention strategy on your data. That means know how long you want to keep it. It could be based on business requirement, regulatory requirement, like figure out a way that your storage will not keep going up and up. Otherwise what happens is even if you start with a 5% storage cost, in year three, it'll be 40% storage cost because you're not purging anything old or you're not archiving anything old. So it's very important to keep a retention strategy. Second is a lot of data that we're loading today in our data warehouses. You have to understand the consumption patterns as well. So there's no loading something in real time when you're consuming it once a month because there is a cost to loading data as well. So these are the insights you want to draw from a tool that allows you to know, okay, what is the best way to load the data that helps with the usage. And the last point is on the environment cost. One of the things we notice is when you're building data pipelines in lower environments, there's a lot of room for efficiencies there. For example, one of the things we did was we made a rule that says you cannot get a in a dev environment, you can't have a server large, greater than small, for example size. And just by enforcing that rule, we saw a significant savings. And if there is a need for something larger than that, you go through a certain approval process. But to enforce rules like that, to enforce the jobs that are when you're testing something, when you're developing something that are not just running, when you're not working right. And those things can happen. And so it's very important that you pay attention to the lower environment, even though that's a much smaller percentage of total cost. But that can also balloon out of control if you're not putting in the same inefficiency checks that you're going to put in production. Right. So point I'm trying to make is don't forget about the lower environment cost. That can sneak up as well.

Yeah, I love this overview and I love how you thought about all of these edge cases and all of these drivers of bullying costs when designing Capital One Slingshot. But maybe here switching gears slightly and thinking about how leaders and organizations think about their data management journey and to avoid ballooning costs, a lot of organizations are still early in their cloud journeys. A lot of organizations are still setting up their cloud infrastructure and maybe what advice would you give them to make sure that they're driving as much ROI as possible when it comes to data management? How do they avoid being in a mode where they're increasing investment and ballooning and costs are ballooning without necessarily driving a lot of ROI from data activities?

There are times that companies that are moving to the cloud don't see what's right around the corner is they go and then they face all these different challenges around governance, around cost, around accessing data. So my recommendation and advice has always been think about data governance, data management. When you're starting off the journey to the cloud, sometimes it's much more harder once you've opened the Pandora's box to put everything back in. Because once it's much harder to do that if you don't have your data registered across your organization now you're going to have to go through and collect all that. But if you think about a central tool, central policy, then from any new data that's created automatically will be well managed, well governed. So that's, that's the first piece that you should think about is invest in a good tooling that allows you to do the centralized policies, enforce the centralized policies, but give ownership to the, to the lines of businesses as well as on the cost side. I would recommend same thing. It's very important to from the beginning put in the right policies on how you're going to do chargeback to your line of businesses. Are you going to have a budget or not? Are you going to have a way to request more funding? All that needs to be part of the data platform. In this modern world you can't just have, since it's a variable cost, you need to have a proper funding, proper budgeting and proper way to asking for more resources. The right approval workflows should be built in and then the visibility is very important on the way you're spending your money.

Yeah, that's. Couldn't agree more. And what's interesting hearing about your journey now, leading Capital One Slingshot, as we discussed a lot of ways, in a lot of ways, Slingshot was built upon capital 11 solving these problems that we're discussing right now, maybe walk us through briefly the journey of productizing a tool built internally and productizing it to the wider market. What changed, if any, in your approach as you were productizing Slingshot?

The first thing, what we had to do was we had to build the tool we built within Capital One was very specific to Capital One. Right. So there was a Lot of hidden integration with capital one infrastructure already. So we had to build a SaaS platform that was supposed to be multi tenant, that is supposed to be multi tenancy. It has to protect the customer's data, it has to make sure that the datas don't overlap each other. So we had to create a whole new SaaS framework, we had to create a different software development practice to help with this journey. And one of the things that you're going to see from Capital One build software is that it's security, performance, scale, resiliency, these things are just built in into our DNA. Right. So everything we build is built for a hardened platform and that's how we've always started. Even though other startups will start with just providing you the features, then figure out how to harden it later, we've always made sure that the product you see is going to be a hardened enterprise grade software.

And as we are closing our episodes today, Salim, when looking ahead, maybe what are trends that you're excited about when it comes to organizations managing their data effectively? How do you think the landscape will evolve over the foreseeable future?

No, I think it's already evolving into that federated mindset. The more companies, they realize that it's going to be very hard to manage it from a centralized point of view. Right. So that's the shift that's happening, but it's not happening as fast as I'd like to see. But it's clearly evident from the strategies of different companies, the way they execute is a little different. They may keep data across multiple platforms, but what's interesting is you have a central catalog across all of them, a central access policy across all your data. And those are the key. That way you break the silos and you have a single way to get data even though it's owned by different line of businesses.

Yeah, that's definitely an exciting development as we see that federated mindset evolving and empowering organizations and different lines of businesses to use their data effectively. Finally, Salim, it's great having you on the show. Before we wrap up, if you have any final call to action or notes to share with listeners before we end today's episode.

Yeah, absolutely. This is what I say quite a lot is when it comes to cost optimization, it's never a once and done deal. You have to be vigilant. It's a constant battle. So you need to be vigilant, you need to be watching for new users, new usage, new usage pattern. Workloads are always introduced in the system. So don't focus on reducing cost, I would say focus on reducing waste and reducing inefficiencies in the system. And then you will be able to scale with peace of mind, knowing that the money you're spending is going to the value generated for the business.

That's really great, Salim. It was great having you on Dataframe.

Thank you so much. It's been my pleasure.