How to Build Secure Data Pipelines with LogDNA and McAfee

Analytics and Data Science at McAfee join us for this session. They will be discussing how to build reliable data pipelines. There will be time for Q&A, so please make sure to drop your questions throughout the session into the chat below and unmute yourself and turn on your video to ask Tucker and Stacia live during the Q&A portion. Tucker, Stacia, take it away.

Thanks, Brian. So just as Brian mentioned, my name is Tucker Calloway. I'm the CEO of LogDNA. I've been with the company for about 18 months now, and prior to that, I've spent the last, I'd say, probably about 8 or 9 years in DevOps, so just quite, just a little bit before DevOps was really a big thing. And so then focusing on DevOps has been a big part of my career, and I'll let Stacia introduce herself.

I'm Anastasia Zamyshleiva, VP Data Engineering, Analytics, and Data Science at McAfee. This is a very interesting role where the goal of my team is to empower employees to excel at work every day through the data. Also, we are focusing on the consumers where we provide the better protection, better service for our customers by using ML and AI. Prior to McAfee, I had huge experience in building data management platforms. In 2011, together with Manish Sud, I co-founded a company called Reltio and helped to grow to an award-winning SaaS company. And there we were driving digital transformation for enterprises by providing single source of truth for the data. I'm very excited to be here and talk about my experience with data.

Great. So just a quick overview of the agenda for today. We have 5 main topics. We want to give a quick history of data, talk about some of the trends we're seeing today and how to build reliable data products, and then we'll focus on the leader's role and how trust plays in that as well. So thanks for the introduction, Stacia. I certainly have a long history there. Can you maybe describe for the group How have you seen the volume and complexity of data that companies deal with change over the last 5 years?

Yeah, thank you. The volume of data is growing exponentially. But I'd like to deep dive and discuss the reasons behind it. First, with democratization of cost for the storage and for compute through the cloud, now we can pay much less for more. And organizations started to instrument collection of more and more datasets. Resources that potentially will have chance to be used in future. The second driver for, for that increase is that look at the systems, they're having increased requirements for availability, for performance, for velocity releases. This requires a new level of monitoring and logging that allows to identify by different signals that, hey, there is going something wrong with a system and trying to predict some of the failures and address them in the shortest time frame. But I really like the last reason. This is the value. I see that a lot of enterprises start getting the first major wins under the belt, and this is really exciting. The biggest contribution to this, I believe, is decreased expenses. So when we are making our employees be more productive by reducing by different tools and data. We are also reducing the cost of acquisition or retention of customers, cost of goods. But even further, through the data, companies often identify new avenues for innovation or for disruption. So we are discovering new segments of customers that we were not penetrating well enough before because we didn't have good understanding about them. Especially this is true for consumers. Or even we are identifying unmet need where we can launch new products. And as we're launching new products or existing products, there is very fast launch with customer obsession when you're defining the metrics where we want to trend. And you are using this build, measure, learn about customers to make sure that we are building what customer is actually looking for, and that is that is addressing their needs. So to summarize, the top sources for contribution of the growth, this is applications with telemetry, cloud computing, especially after all the story with microservices, and development and adoption of more technologies around AI and IoT.

Thanks, Dacia. I mean, we obviously see that in our business at LogDNA and hear that from our customers. I think one of the key themes that we hear often is that so much has changed in the last 5 years even, that the systems that were put in place 5, 10, 15, 20 years ago, they can't scale to both the volume as well as the application of value that you mentioned in data today. So it's an opportunity for— well, a lot of exciting opportunities. So with those opportunities, what trends do you see? How do you see things evolving? With data today?

I'd like to represent the trends of the data through the lens of 5 Vs. There are different variations of Vs for the big data, but I really like this, this approach when we talk about volume, velocity, variety, veracity, and the value. Volume is growing for sure, but what is becoming even more critical is the velocity. Real-time is a new norm for big data. We are using speed now as competitive advantage to make fast decisions. We are getting more different data sources and more variety of the structures that we have in our data lakes. And think about all the organizations that now are adopting graph structures to derive insights. This is, this is really impressive and adds more to the variety. Veracity equals to quality. Garbage in, garbage out. You may be wasting your money by not getting the relevant insights because of the poor data quality. And the top of this pyramid is the value. This is the biggest P for me. If you're not getting the value from the data, then you just and scratch all the previous ones, you're just wasting money. You really need to focus on how can all of these bring the business outcomes. And as we are going to that kind of world, this requires changes not only in the technologies, in the tools, but this is new processes, new ways how do you organize— how do you build organizations, new types of roles for the team structures. I feel that with my past experience in applications, I feel that data is going through the same path as 7 years ago we were going with microservices. So we are breaking the monolithic data lakes into the data domains. And we started to treat those business data domains as products. We have the owners of the data who understand not just some piece. They understand this whole domain, how it is being used in and out. And those data domains are focusing on the customer. How we can drive the adoption of those data domains and how we can bring value to the customers so they'll be using those. So this kind of approach is changing how organizations are switching from what my gut feeling is to what data tells us. This requires a very different velocity of the changes, even in the data world, not only to the pipelines, but also to the outcomes. And what we were doing in applications in this case with microservices, we were adding a lot of automation, testing in the release management through CI/CD, testing in production, be able to roll back if something goes wrong. So the whole new area of data observability tools appear in the data world that, again, is very similar to what was happening in microservices. I went through that path in applications, and I'm really excited about the opportunities that we'll have in big data by focusing on the value and the business domains.

Yeah, I think it's exciting. I really love how you take a construct or a premise that's been— that we've been through before and a transformation we've been through before and apply that to data. I think that informs us a lot on the opportunities with data, but also some of the challenges that we'll all face organizationally, technologically to go to do this breakup of these kind of monolithic data structures into the more domain-specific. So it's an exciting opportunity. It will also be an interesting journey for all of us to make or focus on this domain. So as you think about that, what tips do you have for us? What are the key considerations we should all be thinking about when it comes to building reliable data products? Because as we're focusing more on trusting those— and I'll talk a little bit more later about my emotional decisions versus my data decisions. As we trust it more and as we build more reliable products? What are the things we need to be considering?

When we talk about reliability, data products and company can fail on the business and on technology side. In order to remediate the business side, and this is something that I would suggest to address first, is we can use some kind of strong structure and thought process. And by the way, I can't express how many data initiatives are failing because of not enough to the business and bring the value. So this is why when you think about the data initiatives, you start with company top initiatives. What are the KPIs that we are trying to address? What are the driving initiatives and how data can help? After that, you are building data roadmap with focus on users, on end customers, how you can help them. You define requirements around your customers, and you're defining what success will look like, that you constantly keep measuring. What are the KPIs? Are we moving with our data products in the right direction? What is the number of active users, business impact, revenue, cost reduction, customer satisfaction, and all other measures that are relevant for you? So you're assigning owners and making sure that you are using data management platform to accelerate this whole journey. And the governance model to make sure that all data products can play together and not be silos. And when we talk about reliability for the technology, I would use the same approach as we have in applications with SRE model, focusing on reducing time for mean time to detect, mean time to resolve, mean time between failures by having the automation for testing, for release management by easy rollback. So again, I would like to highlight that for reliable pipelines, you need to focus both on business and technology. And Tucker, you being CEO and being a leader of the organization, seeing so many implementations of the data, what do you think is the role of the leaders building these reliable data products?

Yeah, I think, well, I think you started the conversation and say that it's key that we buy into it and we believe in it. As I referred to earlier, I have a tendency to make more emotional decisions than I do data-driven decisions. I've kind of learned over time that I need to balance those two. And in doing so, what I find is if my emotional or my gut opinion, as Stacia referred to earlier, matches with what the data is telling me, then I know— then I have a really strong feeling that we're making good decisions. But then what you have to do, especially if a person kind of has my orientation that way, is when the data is not telling you what your gut's telling you, you have to have the courage to not make the decision even though your gut's telling you something or your emotions are telling you something. So relying on the data is very important. But that, of course, that really comes back to common definition, making sure we're looking at common sources of data that we all believe and trust in the data we have. And so trust more than anything, I think, is what it comes back to, what it comes down to. Like, if you're going to really make decisions, if you're going to really standardize and also not overcomplicate, you just want to have a very simple set of data measures to drive your decision-making process. But if you're going to do that, you have to have a super high degree of trust.

Yeah, and trust is such an essential thing for the data. So when you really need to focus on the trust of the data, starting with having the common vocabulary and common single source of truth that is aligned across different departments. So the meetings about decisions on the company level will be not about, hey, my data, my numbers are better than yours numbers. It will be about the making decisions based on the data. Also, for the trust, we need to consider and put as essential part security and make sure that all the sensitive data is protected from the start. It's impossible to gain trust and become data-driven organizations if you don't show the value. Value is critical part for having the trust. And constantly deliver certain milestone and celebrate wins on the company level to gain that trust for the data. And through this, you'll be having more and more adoption of the data. People would like to use it more frequent basis. So this is when reliability comes and become a bigger part, making sure that you're defining service level agreement for your products, for your data products that are driven by what is the criticality and what's the business value, making sure that you have your data availability, freshness, level of quality, quality and defined, and you're keeping measuring and keeping make sure that you're achieving those goals.

Great, thanks. Yeah, so if I, if I look at our, you know, excuse me, my top takeaways here, it's, it's on the topic of creating reliable— I'm sorry, reliable data pipelines. It's really that you have to think of it as a product. I think that resonates with probably most people on the call. Most of us are focused on products in general. And so applying that thinking to data and not thinking of data as an output of a product but actual product on its own is super important. Using data to drive business outcomes and having the courage to, to not just go with emotion but to go with what the data tells you is extremely important. And finally, focus on trust. And one of the things I love about everything Stacia said today is really taking, as I mentioned earlier, really taking just general concepts like the what she laid out for us when in terms of creating trust is I think a way you would apply the creation of trust in almost any situation in your life, but applying that specifically to data is super helpful and such a foundational thing to creating reliable data pipelines. So thank you. I think we'll take questions now. Brian, is that right?

Yep, we can go right into questions. If you have questions, by all means, please go ahead and put them into the chat. And Tucker, I'm starting to see some come in.

OK, great. So the first question is, how do you address or prevent confirmation bias in data analysis? So Stacia, do you want to take a pass at that and I can take one, or do you?

So typically, you start with defining what's the common vocabulary across different stakeholders and make sure that you're using this common vocabulary across the organization. And then— making sure that this vocabulary is being used across organization. Sorry, a little bit— I'm looking right now.

Yeah, okay, I can go too. So mine's actually related to Rickys, I mean, I have a lot of personal examples where we have different datasets telling us different things. And so what happens oftentimes— oh, there I am. I think I came back. Sorry, I got hung up. What we often have is like we might have competing datasets and one tells you one thing, another tells you another thing. And so like, how do you avoid the confirmation bias of, hey, when one tells you one thing, then the other tells you the other, or you have this temptation to go out and find another thing that will tell you what you want to hear too. And so I think what Stacia is saying is go back to the vocabulary, you go back to the trust, you go back to standardizing on what that KPI is. But you have to commit to it and believe in it. And you have to adjust it, of course. If there's a defect in a product, you fix it. Same with the data. But I think you have to really standardize and commit to it.

Yeah, in our case, what we are doing— there could be different data groups. Making sure that you are aligning, making sure that you're sitting together. There could be different definitions, and you're coming together and getting to what it is. What is the single source of truth for the data? What is the data quality level? What are the definitions that we are using? At some point, you'll just need to cut that knot.

Yes, we have another question that came in too that said, or a statement, to have success in a data-driven project, the leader will have to believe in the data and understand it. What do you think? Clearly agree with that. And I think, again, going back to treating it like a product and testing it and deploying changes and things like that, I think that's where that thinking really comes in. You have to have that discipline. And so many times we create a report and then we just leave the report as a report or something like that. We create some dataset and we just look at that dataset, but the ability to test it, to watch it change over time, to treat it like something that's important to be observed and apply observability principles to it is super important.

And what I find usually is that data culture comes from top. And so you show that picture where we are all heading through the data. And then it needs to be delivered when all of this data culture is being accepted and being promoted from the bottom to up.

Great. So—

Great. I think we are at time. But that isn't to say that questions can't be answered. They can be answered. And if you can all head over, LogDNA will have a room together, a lounge together in their booth so that you can ask those questions there. But thank you, Tucker. Thank you, Stacia, for that great and insightful session. We're going to have a lot more great content still to come today, so stick around and see you all in our next session.