Getting ROI From Your Data Literacy Program | See Results Clearly

Over to you, Susan.

Great. Thank you, Richie, and want to give my thanks to Data Camp for inviting us to join this session today. Super excited to talk about lots of new and different things on this topic. And also just a general thanks to Data Camp for all the great work that you guys do to continue to help every data leader, L and D leader specifically involved and responsible for data literacy to really deliver the best program that they can for their organization. So hopefully, as you've joined, you're taking a minute to put your name, your role, company that you're with into the chat and a little bit of what you would like to get out of this session so that we can try and add comments that are a little bit more relevant to you as we go. All right, so we do want to make this a little bit more conversational, even though it's in chat. So let's get started with just what's your biggest challenge related to data literacy and the data literacy program that maybe you're trying to get it started. So in the chat, if you would either put in something like 1, 2, 3, or if your answer is 4, maybe just 2 or 3 words that quickly summarize what your biggest challenge is related to your data literacy program. It'll take a minute for those responses to come through to us. But so maybe you're trying to get funding. Maybe budget approval is currently your biggest challenge. Maybe having a very clear understanding of what your data literacy strategy is about. Really trying to still get clear requirements for stakeholders. A lot of times we're kind of given the mandate to just go, just get something started. But eventually having a clear strategy and direction comes up in conversation. Or maybe it's specifically on our topic today about showing ROI and how your program can impact business results. So take a minute, if you would, let us know what you're thinking. And Bianca is going to take a look at some of your responses as we go through.

All right, some of the first responses coming in. I'll show a few on screen. So Shabazz says two and three. Oh, Julian's showing us details. No more responses yet from the audience. We'll give it a few seconds for this more to come through, but don't be shy. Okay, we've got another one there. Swami Nathan is saying 1 and 3. 3's most popular so far, but pretty small sample size. Who else? Julian's back. Oh, number. Number two as well. All right, Yukian, number three, an anonymous LinkedIn user, says number one. Pretty broad, broad selection. Seeing a few more twos and Threes in the. In the results. And we've got first custom response, building confidence in understanding data and being brave enough to use it. That's another number four from Ada saying building a roadmap for data literacy. Interesting.

Ah, all right, Very good. All right, Well, I tell you what, you're. You're all obviously in the right place today, so I'm glad to say that we're going to touch on several things that relate to the biggest challenges that you have. Obviously, ROI is our topic today, but I think you're drilling down into some other aspects of maybe what ROI can do for you. So what we really want to see you guys get out of this session today is a little bit, maybe a different view on how to deliver ROI at several. From different aspects of your program in data literacy, how you go about doing that. What we want to share today is this BADR framework. So BADR is an acronym, as you'll see shortly, in how to actually go about showing ROI for your program. And then finally, it's not just about doing it once. Right. How do you continue to drive roi, scale it, and through that really drive adoption? One of the things we're going to do today that I hope is a little bit different from what you might be expecting is we're not going to look at the L and D process in this conversation today. We're going to come at this from the point of view of the business and hopefully add to your thinking and understanding about some things that you could do to enhance your program. So the way we're going to go about doing this very quickly generally is going to be, let's take a look at kind of the current landscape, what's happening today, particularly in L and D, and how we're looking at and thinking about roi. Then we're going to take a quick look at, well, what's changing? What's different about particularly the R in roi? What does return mean, particularly from a business standpoint? And then with that, if that's what's changing, what are we going to. What can we go do? So we really want to take some time to give you some concrete ideas and examples of ways to begin to implement ROI or go after it differently. And then finally we'll wrap up with a few things that we've learned along the way. Before we launch into all of that, we want to take a minute just to talk a little bit more about our background and pianca, why don't you go first?

Yeah. So great to be here. I'm Piyanka CEO of EHRing and my background has been in data science and analytics for two decades. My master's thesis took more than two decades but my master's thesis two decades ago was in AI. So that's how far I go back in the world of data and statistics and then with companies like Adobe and PayPal, I really learned how do you put all of this data and algorithm and statistics knowledge and apply it to the world of business and drive impact? And that's part of what we have been doing at ehring we have two sides of our business one in one. On one side we work with large enterprises and drive large scale data enterprise projects in driving impact, measurable impact, ROI and dollars. With our, with our work as well as on the other side we are sort of McKinsey of startups for all things data. So we do strategy and execution, data science, analytics, engineering, engineering, data engineering for our growth clients.

So Susan, yeah, and just excuse me, still fighting a cold. I think the viruses are going around worldwide just a little bit more about my background. Having been an L and D for quite a number of years, what I've really enjoyed and appreciated about the opportunities that I've had is I really talk about my role and my experience more from an operations standpoint, solid line reporting into either a GM ahead of a business unit, a chief technology officer versus kind of a solid line into to hr. Which means that a lot of the mandates that I was given have been directly aligned with how is learning going to make a difference in the bottom line of the business. In addition to that, my background is data has been the common thread throughout, whether it is in continuous improvement as a lean six Sigma master black belt or in L and D roles where you're really shaping people as well as practices and processes. The ability for L and D people to have a good command of data and business data has been a huge driver of my career in the past. And so in addition to some of the other opportunities I've had in certifications, I'm also a certified citizen data analyst and that's an area that I continue to practice in as we work day to day with clients. So yes, I am a data nerd just like a lot of folks that are out there. And I know we've got it looks like almost 300 people watching on the line. So hopefully some of the L and D community that's on the line today can relate to some of the experiences that I have had. So I'm a huge believer obviously in data, the value of data and the Importance of showing data in addition to return on investment. So one of the things that, and again, this isn't just for us. I think it's great to share with each other kind of the small community of people that we have in this, this webinar to let each other know what's going on in your world with respect to your data literacy program. Curious to know what's your motivation? If you think ROI is important, let's get more personal. Why is it important to you in your particular role? If you have ROI data, what are you going to do with it? How are you going to use it? So we've teed up a couple of potential answers here. Again, if you just want to pop a 1, 2 or 3 or 4 into the chat, we'd love to see what you guys are thinking and share that with everybody else on the call. Again, if your biggest challenge is, is to get budget, is roi kind of that key that's going to help you unlock maybe additional budget beyond just maybe a proof of concept or pilot that you've been doing. A lot of times what we see is ROI data being used to show truly that a DL program, DL stands for data literacy, that your program is working, that it is effective, or maybe it's your way to be seen as more of that kind of true value added partner for the business versus training fulfillment. Maybe it's something else. Again, we'd love to know what you guys are thinking.

All right, we're starting to get the some responses back. So Matthew says one and two. Alex says three. We've got a complete set now. Amaranta two with three exclamation marks. Obviously that seems perhaps the most important there, but a long way. Daliana says three. And oh, we got a custom answer from Ada. So ADA saying making better decisions for the next steps. Okay, thinking a few steps ahead there. That's good. And then we've got Joel says 2 and 3 and anonymous LinkedIn user, custom answer saying it's important to demonstrate the value of using an improvement approach. Also talking about like a, I guess, continuous improvement here or. Yeah, finding some benefit.

Nice.

Okay. And the answers keep coming in.

So lots of 1, 2 and 3.

Swami Mason says 1, 2 and 3. These are all important. I like it.

All right. Bianca, did you have something?

Yeah, I said it's heartening to see these answers that ultimately, at the end of the day, all these are very important. And these are different ways in which you show up as an LND leader to kind of show your value to the organization for the Work that you're driving.

Yep. Yeah. So tactically, getting that budget approval is usually top of mind. But again, what are you going to continue to do with ROI data as your program progresses and as it matures? You'd be surprised. A lot of times people don't know the answer to that question. They just think, I need ROI data, but I'm not sure what I'm going to do with it. I think what you guys are articulating and we love to see, is that you're going to use it to influence. And that's exactly, I think the right mindset to have. Another aspect of getting ROI data is what. What kind of. How do you kind of categorize return? What does return mean to you and your data literacy program? What are you actually tracking or maybe you're not. What would you like to be able to track? Is it more along the lines of kind of hard dollars, revenue, cost margin, improvement? Anything that you can really put a good hard quantification to? Is it productivity? More productivity related? You know, and we kind of define that as, I can get back some time, I can do something else with that time. Kind of more at the individual level. Sometimes it's more about efficiency, which is more along the lines of creating resource capacity that if we can free up some time, it allows us to do a lot more with that capacity that we now have. So kind of taking productivity and really scaling it up. Or again, maybe it's something else. Curious to know, kind of. Where are you?

All right, we're getting some responses back. So custom answer here saying, improved outcomes for people serving the nhs. Okay. And Julian says two and three. I've seen another three from Matthew. Ada says two and three. Number one, surprisingly not showing so far.

And.

But we've got a custom answer saying being the first to build the data culture followed by data strategy. That's an interesting one.

Yeah.

Oh, and Amarantha says, looking for ways to increase revenue and save costs. I think a solid vote for number one there. And from Tanya, we have data around compliance. Ooh, compliance is an interesting angle here.

Yeah. So we often don't see very many wants. And, and I think there's a. There's valid reasons for that. And we'll go into a little bit of that discussion kind of, again, looking at the current landscape, I want to kind of quickly go through a few of these, wrap up some of these questions. So what is your quantity? Quantitative data, Quantitative ROI data. What is the roadblock to getting quantitative data, if you have one? And again, these are These are things that we commonly see or commonly hear in the current way that ROI is being handled. Maybe you don't know how to align what your data literacy program is doing to quantitative metrics. What I often hear from L and D folks is, you know, it's just super complicated and it takes, you know, a resource commitment to get the data. It's manual maybe, or, you know, training happened a long time ago and now we're waiting to see if there's some. Some sort of impact that we could collect, but it's very complicated to get. A lot of people will. Will point to learning technology as a roadblock.

Yeah, there's quite a few of number two here, but there's also a fair bit of one and three.

Yeah, we got all the numbers. We got a two. Yeah, one to three here.

Yeah. And how many of you, particularly in L and D, have been in conversations about, you're presenting the impact of the training and somebody at the table is going to say, yeah, it wasn't really the training that achieved that. I mean, there was coaching that happened afterwards. There was all of these other performance management, all of these other factors. So it really kind of sometimes is hard to isolate the impact and measure just that. Right.

Yeah. We hired a new team here, we acquired another company. All of these macro business dynamics fudge your. Whatever impact you have attributed to your program in whatever meaningful way.

Yeah. So let's push forward just a little bit again in the current landscape, what we tend to see, and you're echoing some of these things, is in the absence of our ability to get really hard quantitative business metrics, we attempt, we do the best we can with what we have to show. Roi. I call these kind of proxy metrics. You know what a proxy is? It's something that stands in for the real thing, if you will. So a lot of times we'll see. We've got a competency framework, we've got Personas that we're training to and we're showing progress on some goal or some target that's been set, or we're measuring skills. And, you know, there's just enough consensus in the room to say, yeah, there's some correlation to performance. We, we can't really quantify it, but we'll just assume that it's there. And so here's our skills development data. Sentiment surveys are very common as well. And showing progress on. Do we feel better about the decisions we're making? Are we making more decisions with data? That type of data? Yeah, it's quantified but again, it's coming from kind of more qualitative sources. Same thing on productivity, getting hard measure on productivity. If we can't get that often, it's replaced with do people feel like they're being more productive? Again, nothing wrong with these measures. The challenge is when you sit down with executives or business leaders, decision makers, which is the world in which we live every single day, whether we're talking about data science initiatives or data literacy and training people, they're wired for two things. The first thing is, and anytime we present data, certainly to them is how are you helping me solve my problems? That's, that's number one. Number two is who's accountable? Okay, I can absolutely guarantee you 100% of the time those are the questions that are in their, their head when you engage them in a discussion of a data literacy program, or any learning program for that matter. But we're going to talk about data literacy here specifically. So often when we present this kind of data, kind of proxy metrics, they're trying to answer those two questions in their head because they want to help their decision makers. They want to make a decision. But sometimes our data, even proxy data, you know, you can extent change that word to proximity. Data is so far away from what their business problem is, it's just too much of a stretch for them to really relate to what we're presenting, why we think it adds to business value. So what we want to talk about then is what's really changed and I think is changing very quickly. We literally, like I said, spend every day talking to business decision makers about data literacy. The 90% survey feedback here is actually data collected by Gartner. So it's not ours. This is what executives or leaders say about data literacy programs according to their survey. We can certainly echo this without fail. As a matter of fact, Pianca and I were in a meeting yesterday, just yesterday, with 11 business executives. We weren't asking them about, it was more in the context of getting ready to launch a data literacy program. And we simply asked them the question, what does data literacy mean to you? And business leaders across the board answered, data literacy means that we are driving decisions with data that are going to have business impact.

And they had an exact, exact number. There was exact, you know, a range 2.5 to 5 million impact that we want to drive from that.

Yeah. So if you, if you think about from the business point of view, what do they want? What is their view of roi, if you will? And it is how does they need to see how decisions are being made, they expect a change because right now it's not delivering what they expect. The second thing is they want to see data driven decisions that are aligned with business goals. And the third thing is they want accountability for results. So question is, can data literacy programs deliver that, that business impact? Some of you may be saying absolutely yes, others might be, I don't know, I'm not sure. So we want to go into how at this point, but this is kind of going to be at the core of everything that we talk about for the remainder of our time is if this is what executives say this is what they want, is your program designed to deliver that? So can it be done? Obviously, we wouldn't be here if we didn't say the answer is yes. These are just a few headlines from what we would call our data literacy ROI success stories. We could fill up another page with many more. But rather than go into kind of a case study by case study, we thought we'd begin by sharing what do these things have in common? How did we get there? What these things have in common is in every single one of these situations is there is a business champion, there's somebody in a business role with a business metric that is driving them, that is directly involved in what data literacy means, what success looks like and how to go about it. Those goals are very much aligned from data literacy to directly to the business metrics. So there's no proxy metrics involved. They're very close. There's a cross functional structure in decision making. So if I look at most data literacy programs, and I've done this myself, so I'll talk to myself on this one, a lot of times we immediately structure them around training for the individual. What we know about driving ROI is that in decision making, it is not individuals making decisions, it's a team of people making decisions. And in order to make them productive, how can we help? What do we need to change in the decision making process that's going to make them effective? The other thing that this approach has in common is that we actually can help people learn to become data driven decision makers while they are actually working, working on business projects. At the end of the day, this is what closely ties together the learning piece with delivering a business roi. Because the expectation is that's what they're going to go do immediately with what they learn. And the fifth thing all of these have in common is there's no requirement on technology. We don't have to be in a big enterprise with deep pockets capturing a lot of data for us to then try and correlate into what is the, what is the ROI correlation here. Okay, so we're going to go into how this is done and I'm going to turn it over to pianca.

Yeah. So I wanted to share a few things. One of the things Susan and I both, as we are on receiving end of companies reaching out to us and asking for training, we hear quite a bit, people asking for, do you guys train in Python? Can you help my team learn visualization in tableau charts and pivot tables? Do you do power BI training? Endless. What I wanted to focus on and focus your effort on is that that's all part of the story. If you look at user and system interacting and collecting data and then aggregating and storing it in a dashboard, that's just business intelligence. That's part of the story that still would not connect. If you were to give your, you know, as you think about data literacy and you give those kind of training to your, to your employees, you still would not connect the dot all the way to what executives want from you. At the end of the day, they want dollars, they want to see some kind of return. So there is a process. The analytics, or what we call really, you know, the ability to turn data into dollars is actually what is what your data literacy program should deliver. While we think that many of the, many of the time when, when L and D leaders are thinking about it, they are thinking from a very tactical requirements. You know, they need Python training, they need this training, which is half of the story. It stops at the dashboards and reports. It doesn't go to insights and how it feeds into planning and how it feeds into action and then dollars. So next slide. So there is a need for us to. You go to the next one.

I don't know why this one is here.

I'm sorry, can you go to the next one after that? Yes. And then the other thing that we think about is when we are thinking about data literacy, most of the people are thinking about charts, table, you know, hard facts, numbers. But data by itself is meaningful, meaningless because at the end of the day data turns into dollars because of decisions made by people. So when you are thinking about data literacy, think about skills not only just on the blue track, we call the left one blue track. That is all the technical training and skills that people need, but it's also on the, on the green track of how do you work with stakeholders, what decision, how does, how do business models even work? And you, so you kind of need to combine both the blue track and the green track. Analytics or the ability to deliver dollars from data comes with combining technical skills with these business stakeholder management other skills. That's when it's complete. That's when data literacy is skills as learned or as delivered is complete. So this framework that we've been talking about, Badir is an acronym for these five steps. It starts from and this is again very very high level. It's very hard to understand what may be talking about. We'll just briefly go through some of the details here. Salient point but it stands for these five steps. The first thing to know as people are embarking on trying to convert data to dollars is not the first step is not data, not collecting data. It's about business question. And this is where you start the engagement with your executives and end user. Because you're asking what is the goal? What is it that why are we even embarking on this journey of even trying to what is the question we're trying to solve. And then you go into planning for that which is the second step. Then you collect data and only relevant data. This is a third step, data collection going into fourth step as data driving insights in a systematic way and then making recommendations. And all throughout you have the blue track going which is the technical aspect that you need to do but also the green track in parallel which is how would this connect the dot to impact to dollars to returns. So there are significant efforts. And then this again on the right side is Badr kind of at one level below it's a 33 point steps. There's a lot of details here by the way. Badr was created 13 years ago as I left PayPal and started airing and wanted a way to empower our clients with the know how of how to use data. This is the same framework we use all of us at airing are all of us are botherist. We all use bother in our day to day function solving complex problems. But we also use this to create a common vernacular for our clients from executives to the frontline managers. Everybody then starts speaking the same language. And what this does bade does is it aligns people on a shared goal before ever touching data. It helps people make decisions towards ROI and essentially unites the organization towards one way of looking and moving the ROI forward. Because at the end of the day businesses work and they function because of the cash flow and the revenue and the roi. Right? So sort of all of our efforts are then directed towards that top line that the companies are going for. So the why, why, why people Use Badr and, and who uses Badr? Badr is used at all functional levels in the organization. Susan, do you want to talk about this real quick from your.

Yeah, we, we just wanted to give a, you know, a quick snapshot of, you know, who, who would this for and why. It does resonate with, you know, folks at the top of the organization and then all the way through, but for slightly different reasons. But it all relates back to kind of those three things that we're going to keep coming back to is, you know, what do leaders want from data literacy? What's their definition of what success looks like? And as we see again, people taking all the great skills that they've learned, probably through data camp training, you now know how to do a lot of brand new things, use new tools. You didn't know. Now what, now what do you do with all of that? How do you go really work on important opportunities? How do executives articulate what those most important areas are from a business impact standpoint? We have a lot of, you know, folks at kind of that next level down that feel like they've just got so much data. They don't, you know, where do you begin? Or they've just got so many dashboards that they're constantly, you know, trying to get put up or trying to spin up or, you know, trying to evaluate. You know, quite frankly, there's a lot of effort that goes into creating dashboards. Many times they don't get used. So all the way through, even to the L and D team, we've used it to help really drive out better metrics that are more closely related to business impact. So this model has a lot of application to a lot of different folks. We're going to be able to just touch on it just a little bit. We do want to be able to give you a couple of quick little examples before we wrap up and still leave some time for some questions. But we could easily go into each one of these areas and, and give you lots of good stories about how it's been used.

Yeah. And so we might want to move through this fairly quickly. So I'll talk through this just at very high level. So there are, within this large framework that we shared, there's framework under framework, so it's fairly involved and that gives us the ability to really deliver to its, to its promise. So business question framework really reduces into iterations and it drives the alignment between stakeholders towards one common goal. It really helps with critical thinking. So the next one analysis plan, again, fairly involved process. What it helps is it helps to create a plan which meets the time and scope of what we have agreed on for the business question. And it also, you know, all throughout you might see some things in blue and green. What we're trying to show here is there's. In each of these steps there is going to be things that the analyst and the stakeholders would do towards, towards, you know, towards the technical track. But there's always going to be actions towards the green track, the decision makers, alignment with stakeholder. So it really enables the planning before you're doing so that you don't end up in a. You know, how to get to it. And then the third step is data collection where there's a fair bit of cleansing and validation because you all sort of alluded to it's very hard to get the data that we need. And this by the way is a common, like the conversation we were alluding to yesterday, the workshop that we were part of, Susan and I, that we were leading for our client. The biggest bane that started the first hour of the conversation, you know, 0, 0 minute to the end, the question that everybody was talking about how hard it is to get to data. And that's where our method of how we collect data and how we clean it up, validate it is able to short circuit some of those very big challenges that every organization has. And then the fourth step, the insights is where you, it provides a structured approach and that structured approach really helps you to save time and you use the right method to deliver the right insights because you know, it is complicated and if you wrong, if you use the wrong method, you can arrive at a different conclusion, incorrect conclusion for your business. So you know, right recipe, following a structured method and that leads to impact. Because at the end of the day we are doing this because we decided that there is a problem worth solving for. And then you have to connect it back to. Now that we are looking at some insights, what does it mean for that problem that we started with? And then the last step in Badr is recommendations, which is how we are going to connect. Finally connecting back to the audience, connecting back to your technology, whether you are scoring the databases or whether you are, whether there's a recommendation of what we need to change in our internal process to create probably less friction for our customer and so on, so forth. So if we move on, Susan, you want to talk through this? I think just in the interest of time.

Yeah. So I know we just kind of blew through the five phases of the Badr framework. Maybe just to wrap up with a quick Comment here. One of the things that you can do, we're often asked by folks to say, hey, take a look at our. We have a competency framework, right? A data literacy competency framework or Persona definitions or skills. What do you think about it? Are we missing anything? And what we'll just simply do is literally kind of visually take what they've constructed so far and place it in that Badr framework where it shows up in there almost every single time. We see there's a lot of stuff in the data collection, a few things in the analysis, and a little bit in the last stage of recommendations. Obviously that's where you might have, we're doing a lot of training on visualization, storytelling, things like that. So what, what that visual brings to into the discussion very quickly is look at what you're missing. You're missing the key things up front, which is establishing again the goal alignment, the accountability, the tie to a business KPI. The other things that more often than not are not showing up is how are you driving decision making? All throughout that end to end process, we tend to focus on decision making as got my data, I've got my charts, now I'm in a meeting and I'm presenting. That's where decisions are made.

Susan, I'm still seeing the recommendation. Have you gone to the next slide?

Anyway, my point is what's usually missing is there are so many more decisions that come before that that are typically missed or overlooked in a lot of the L and D models that we see for data literacy. Let me pivot very quickly here and then we'll, we'll wrap up for some questions. So, so where do you start? So we wanted to give you a little bit of an example of where to begin. And I saw a few comments in the chat with respect to, you know, I'm starting with training in my, my data literacy program. Is that the right place to start? You know, where does strategy come in? So this is where literally we start with clients. And even as of yesterday we were asking the question what does data literacy mean to your organization? So this particular client, and this is representative, I think of across the board of most that we work with, they defined it. So we're not using kind of a generalized definition. The first step is for them to be able to define what it means to them and really kind of tie it to their situation and kind of where their starting point is. As you can see, for this particular client, their expectation, their definition includes acting on the data related to top KPIs, speeding up, speeding up decision making as well as the quality of decisions, and again, their ability to have good trusted data and then also be able to learn from it and apply it in a way that helps them continue to improve. Then the second question we asked is, okay, well what does success look like then for you? And these are kind of the classic four things that a lot of decision making leadership teams will come up with. They expect prioritizations from their leaders in terms of what are some of the highest value areas that we should be using our business data, how do we identify them? How do we make it kind of a continuing practice to know what the highest value things are? Then how do we deliver impact from data analytics projects? And we're talking to the business people in this conversation. Yes, it includes some of the data leaders as well, but this is not a conversation that's being directly just to the data team. Okay. And then finally, you know, in this, in a lot of situations, their expectation that projects will be part of the learning, it is part of the way we develop people. It's not something separate. It's not kind of a handoff after the training, not with everybody, but in specific cases. And then finally the expectation that the decisions that are made out of the insights and the recommendations that we are finding that they are actionable and that we're actually taking action on them. And that's what generates the value. The other thing that we want.

I just wanted to add the framework of connecting high value projects to your first of all, identifying high value projects. That framework is called three key questions. It's there also discuss in my book, I think it's in the leadership section. So this book behind every Good Decision, which was mentioned earlier, has a section on how do you identify most important projects for your organization? First, most important metrics, driver metrics, and most important projects. So that method exists out there for you to look at. And then connecting your data literacy program to those projects is how you are going to, you know, complete the circle. So I know we've thrown a lot at you, but there is a lot of reading out there for you as well if you.

So through all different levels of the organization, there are actions that they can take to make sure that there is alignment between ROI and how you go execute on it. The other thing we wanted to share was data literacy roadmap from a particular client. And again, this is kind of representative, maybe a little bit different from the type of data literacy roadmap that you've been putting together. Typically we see ones that show a lot of training related deliverables. So we're going to train this many people, so on and so forth as we go. This is different in that it is designed to show business leaders how we're going to scale roi. Not training, but scale roi. And so in each phase, the expectation is to deliver business value through projects. In order to do that, we're going to train a small cohort of people, at least in this first phase, to get it up and running. And then we're sort of building this model, if you will, of the leaders are engaged, they're identifying what the highest value projects are. We're training folks that are going to go work on that project and then they actually work on it with the appropriate level of mentoring. But the expectation is set at the beginning by the leaders, the understanding, the connection of ROI to business value, the execution on it. And then finally at the end of the the phase, it's set up a quarterly cadence, if you will, operationalizing what are the next projects that are highest value. How did the ones that we just did deliver? Did we get what we expected out of them? So building out that model, if you will, and then from there, if you can prove that model then is delivering on roi, great, go into the next phase and expand it. And then finally the next phase and expand it and scale it again. This is a design for scaling ROI versus a design for training more people.

I wanted to add here a little bit, just clarifying the skills that we earlier talked about. Whether your teams might need skills for charts and pivot table or need skills in Python, all of that is part of this. This is not to say that those skills are not needed. It's about looking at that skill in the context of business and where it's needed, which level needs it, which level doesn't need it, which level needs different. And you can do that when you understand all of your program through the lens of business. Can all of us learn Python? Do all of us need to learn Python? So it's about understanding really what's needed at what level and how does that then connect to the business outcome.

And again, if you kind of look at the scale of this, in this particular roadmap with this client, we're talking about very small scale, one cohort to begin with. So, you know, that's why we're like, we're not waiting for LMS technology or AI to come along for us to be able to prove business value through data literacy program. We're actually going to do it in smaller scale. And at the end of the day, you're probably not going to necessarily have everybody in the organization working on, you know, the highest value projects. But what we can do and what we've been able to do is even through you know, one small team or two or three small cohorts, we can deliver 10x or more business value compared to the investment that was made. Small teams do two big things very quickly. Finally, just a quick what have we learned? Butter is not just something that has to be launched across the entire organization. At the enterprise level, you can start small at a functional level, wherever you do have a really clear business champion with a very clear business. KPI can be part of, it can be part of launching your data literacy program. And in fact that's where we recommend it is positioned so that you can show early success even on what looks like a small scale, whether it's, you know, 10 people, 15 people, that, that can turn into, you know, a couple of very, very meaningful projects that we've seen, you know, return millions of dollars. All right, again, kind of on the presenting either a training roadmap to decision makers. They only see the investment side. They only see the what's, what's, what's happening. If I invest money here versus money someplace else. If I, you know, lose productivity here versus gaining it someplace else, that's just the investment picture. So that's what we really hope that, you know, thinking about how can I present more of a, an ROI based deliverable roadmap that's going to show how we can scale value and the R in roi, not just the investment side. So I think some of these other ones are kind of self explanatory. I will kind of click into number four very quickly. We do see a lot of teams spending a huge amount of time and sometimes money on trying to detail out what are the competencies, what are the competency behavioral descriptions, what are the skills and a whole lot of effort that goes into mapping those kinds of things. And that's fine. There is nothing wrong with doing that. We think there's a way to get started a lot faster before you have all of that really kind of sorted out. Plus, I think there's a value to investing that kind of time and money. When you really have the ability through learning technology to do something with that data, then that's actually going to translate into direct business value. So a lot of things that we've learned, these are just a few things and would encourage you guys to think about some of the things that we've shared today. Take a Look at what is your own understanding of your key business metrics and what is driving them. Have you had that conversation with even functional leaders to say what does data literacy mean to you? What does success look like? What's lacking? It's as simple as that in terms of starting the conversation. At least that's what we have found. Like I said, we've thrown a lot out. I've shown you two things in terms of examples that are typical of clients. Those come out of what we call a data literacy playbook, which is really a very in depth strategy document. We've learned a tremendous amount just over the past couple of years specifically about what really works. And our we're game for sharing more of that, not just those two slides and really would like to expose that. I think it could be a very helpful benchmark, if nothing else, for a lot of folks that are in charge of the data literacy program. So we're inviting you to join us in another session because we couldn't cram everything into here. So feel free to email me if you would like to join that session. Don't have it scheduled yet, or any other questions that you have about anything that we've talked about today.

And Susan's email is susanaering.com There is a. So susanaering.com will type it on in here as well. There's a question here Susan Amarantha is asking what are the usual business questions that ignite a data literacy program and metric. I was wondering, Susan, I might just share the screen because my, my slides from our client yesterday is open.

Yeah, sure.

I might just show an example of our of sort of how you can think about this. So let's see if you're able to see my screen. I am going to open. This is so this is out of context. It's not, it's not yours. But this is one of our clients and this is this was abstracted enough that I thought we could share this with you. So this is an example of how you would start from, you know, understanding for this organization what are their what is there. This is called KPI metric tree. We also call it KPI DuPont again discussed in my book Behind Every Good Decision in the in the it's called chapter is Analytics Agenda. But anyhow, so this, this is the top metric and these are the driver metrics. Dotted line, solid line. When you start understanding these metrics, you create a dashboard which I have we have not visualized here for you in the context as you start understanding what are your drivers of your Main metrics. So whatever your metric is. So some of them could be for, if you're working for a mill, large mill production efficiency, for example, might be a metric. Different metrics for different organization, thereby different driver metric for different organization and thereby different projects. So in this example, what we found was these six projects were found to be very, very important. Their customer satisfaction CSAT was X percent. It was believed if it goes from X to Y, it'll have a direct impact on revenue and thereby direct impact on margin that they're tracking. Same for churn model activation and so on. So this is just one example to show you how you would take, take a set of like take basically an organization, create a metric tree and identify top projects. Now that we have these projects identified, then it goes back to what Susan, if you want to share real quick, again, what we were, what you were sharing in terms of the, you know, the, the cohorts that, the cohorts which are working on projects. I think you had that in one of the slides.

Yeah.

This is how you're going to connect it. I think if, if it makes sense. I know, I know this is a lot and I, you know, Susan is going to go through if, if you were to sign up for the Playbook webinar. She, she goes into, we are able to go into more detail. But this, when we are talking about, you know, review of project roi, ideal metrics. These are the projects that these people that the early cohorts that you have defined are working on and thereby direct impact to your data literacy program. There's no question at this point because this cohort then works directly on projects while getting the training and getting the mentorship to actually execute. And then there's a very clear delta because this is not the first time they've attempted projects like this. There's a very clear delta of, listen, this data literacy drove value for us right away. I hope that answers the question. I don't know how we are doing on time.

Richie, we've got a few minutes. We can perhaps run a couple of minutes over because there are some great questions from the audience.

I think just to finish up Bianca's thought. Which metrics really ignite the conversation? It depends on who you're talking to. Are you talking to the coo? Are you talking to the head of hr? Are you talking to, you know, so each one of them might have a slightly different answer, but ultimately, hopefully all of their key metrics drive up to, you know, a key business strategy somewhere.

Yeah, I have to say I do like your idea from the framework, where you should really think about one of the business goals before you start the training rather than at the end of it. You know, oh, well, did we have an impact on the business afterwards, it's easier to think about beforehand. You're going to be able to measure it much more successfully. All right, so next question comes from Swami Nathan saying, am I right in saying that launching a data literacy program is the first step in framing the data strategy, which is the foundation for any related data transformation? So do you need your training program at the start, like before you come up with the data strategy, or is the order different?

We don't start with, we start with the leaders again. What does it mean to them? What does success look like then really key in on our first steps are going to be meeting them in terms of their definition of what success looks like. I can tell you those three things that we put on that one slide about what they say that has not changed and I don't know how long yet. We build all these programs and metrics around data literacy that, that don't address it. So I would strongly encourage you, number one, is to pay close attention to those. Go talk to your leaders and see if you get the same answer. I think you're going to.

Yeah, sounds like I want to add real quick. So the data strategy is a very important strategy on how you're going to convert data to dollars. So Susan is absolutely right in terms of, for your data literacy program, you don't need to have your data strategy figured out. However, as you embark on your data literacy program and the way we do it, we start identifying metric trees and projects. As your cohort executes on that or begins to execute on those projects, that creates your instrumentation needs and that starts informing your data strategy, which is, listen, I can't boil the ocean. I can't get all my terabytes of data in my. Maybe I can get that in data lake, but I can't get it in my data warehouse. It's not worth spending time to do the physical and logical model and architecture and etl, the entire data that you actually get a dump of. However, because now you have identified top projects and the data requirements from the top projects. Now I know my top 300, 400 metrics, which are most important for me, which solves 80% or 90% of my business problem. That should go into your data strategy saying, listen, let's create single source of truth for at least these 400 metrics. And all your downstream projects are going to go 4x10x faster because now you have a single source of truth. So the process actually goes as an input into your data strategy, not as a, not, it's not dependent. It's a parallel process that starts.

Yeah, okay.

Yeah. So it seems like it, the order was backwards. You need to come with the plan first and then you start your data literacy program after that. Once you have your plan. All right, next question comes from Puneet saying, how did the companies in these success stories isolate the benefits from the data literacy program? That seems to be the tricky part. Right. It's like, you know, you've got some benefits, but actually attributing it to the, the training is harder.

Yeah. So we don't view training and action on the training as two separate things. It's one. So if you. And it's learning how to use data in decision making from end to end, from here's the problem to analyzing the data, getting the data, analyzing it to insights, recommendations, it's from beginning to end. And when we get to that end, point of here's what we found about our hypotheses through the data, and therefore we should go do those things that tie back to the business problem in the first place. And the potential value, that's where we show as people have trained, as they've gone through this experience of using this decision making process together, they're delivering that value right there, right there on their project. So it's. Training's not separate. It's all happening together, if that makes sense. Almost like real time.

Yeah. Actually, I have a question. So the example, one of the examples Bianca, you showed from the slide was about like, retention. So you're saying, okay, people get trained on this, how to improve things. You try a new project and then if retention gets better, you can say, okay, look, people have been trained. What happens if retention goes down just due to other circumstances? Is that the point where L and D just quietly back away and say, people, but there's other problems?

Yeah, yeah. So, so, so retention went up or retention was improved. Sorry. Yeah, retention was improved, but so did the skills of the people who worked on it. And they don't, they won't lose the skills. So they, they continue working on other projects afterwards. This was one, one where they worked while, while they were learning a new skill. So it's very hard to learn new skill and apply it right away, apply it without a mentor. So what we do is we have them learn skills, but then we have them apply with us as the mentor. That's what it gets in their system. Bather needs to be in their system versus yet another training I went to, which I forgot in three days. Now it's part of their system because it's part of the project that they're working on and they see end to end, it kind of gets internalized. Maybe I missed the question, but I thought I was. I'm trying to understand whether.

Oh, I think I was just. I was slightly concerned that if you start attributing any business success from the projects that people got trained on to the training, you say, okay, retention went up. It's definitely because of the training. If it goes. If the business metric goes in the wrong direction, then you don't necessarily want to.

So I think that the subtlety here is that the way in which we should be thinking about data is we start with hypotheses. This is not just trial and error. It's a hypothesis. Our analysis of the data should either prove or disprove that hypothesis. If we thought something was going to improve customer retention, but our data says it's not, therefore our recommendation is don't go do that. So, yeah, sometimes the answer is we're not delivering value with these recommendations because we've shown now through data that's the wrong approach to take and that's a win as far as we're concerned. Now we can go visit other hypotheses that we didn't investigate to see. All right, well, what is going to work?

All right, nice. So let's do one more question before we wrap up. So Albert asks, in terms of identifying how decisions are made in my organization, what's your suggestion about whom I should work with? So if you're an elderly leader, like, who do you need to be collaborating with? Does it need to be human resources planning, monitoring? I guess that's like project management. Who should you be collaborating with?

I always have a strong bias for the business people. And unless your project is purely focused on a human resources activity or key metric, then, yeah, go talk to hr.

Is it worth talking about the typical organizational structure of where L and D sits then? Because I know it varies a bit from organization to organization. Just want to give.

Varies all over the place. No, I don't. I don't. I hope that alignment doesn't matter. Reporting lines don't matter in some companies that they do. It's. It's just the way things are. But as L and D people, as leaders in L and D, the closer we get to the business, whether it's funk, a function could be HR could be still closer we are to that business leader who is looking for decisions to be made delivered differently, goal related with accountability. That's who you want to start with?

Yeah. A problem could be employee retention, employee attrition.

Absolutely.

The problem is, hey listen, our employee churn is extremely high as compared to our competitors. Then that's the business, that's the. Whoever owns the P and L for that, whoever has that problem is your stakeholder to work and then probably your HR analyst are part of your cohort as you're running a training program or a data literacy program for your HR organization. So it could be functional that you're running and you're starting a functional or even could be like we work with one of the largest telecom, we've been working with them on the entire 2000 people customer support. So it could be a business unit, it could be a function that you start with as you begin proving value or it could be wider organization throughout. But the use cases are by your functions, by your business unit. Where are the use cases? Where is the problem that you want to actually solve Using data. And that's where data literacy program needs to fit in.

If I could take 30 more seconds, maybe highlight this question just a little bit differently. Let me give you a real situation in terms of decision making and what you're kind of looking at or looking for. And Pianka, I'm thinking of one in terms of product management, product launch didn't go well. Everybody got into a product manager, got her into a room and said we've got to turn this around immediately. You know, adoption is, is, is trailing so fast. So you've got people from customer support hearing the, the issues and the problem. You got people from engineering and they're bringing their own data into this meeting. This is my data, this is what I look at. Right. Everybody is doing that. What happens is data is so siloed and isolated in our thinking and our actions and it stays that way in our decisions. What we want people looking at instead in that meeting is I'm not here to point a finger at everybody else. Our problem is this, Our problem is getting adoption in this product immediately or we're not going to hit our revenue targets. Bring me your data. But instead of saying not my problem, I don't have any issues, we need to start looking for what are the hypotheses about what's, what's causing this very quickly and then testing that data to say okay, we did see us, yeah, we saw a spike in customer complaints, but is it really driving the problem that we have. So changing our mindset about how we actually use data in decisions like that is really paramount. And if you walk into any meeting today, I bet in a lot of situations you're going to see this isolated kind of mindset in the way they talk about the problem and what to go do about it. Hope that helps.

I'm hoping that's answered the question because we are. Well over time now, it's fine. That's brilliant. So, yeah, hopefully everyone's got a bit of an idea about how they might start thinking about measuring their return on investment for their training. Before you all dash up, I just want to say we've got some upcoming webinars. So tomorrow we've got a session on using Power Bi. Next week there is a session on Wednesday about data storytelling for your data portfolio. And then in April, we've got a lot more sessions coming up as well. So thank you once again, Susan. That was fantastic. Thank you, Bianca. Really informative stuff. I very much enjoyed the session. I hope everyone else in the audience did, too. Thank you to Reese for moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today. I hope to see you all again in future webinars. And don't forget, next week we've got the Radar Analytics Virtual conference that's going to be on Thursday. Please do sign up for that. All right, cheers. Bye.

Thank you.