Analyzing Credit Scores with tidymodels in R

Sam.

Hello everyone. Thanks for joining today's Live training. My name is Rhys and I'm your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to join. But in the meantime we'd love to hear from you. So let us know where you're joining from in the question box. And yeah, tell us what you'd like to get out of today's live training. Just to note that today's live training will be hosted on datacamp Workspace. So if you don't have a datacamp account yet, make sure you register for your free account. I will post a link in the chat now. But yeah, we're using Data Camp Workspace today, so we'll be also be posting a link so that you can code along with us. If you're already set up, feel free to scan the QR code in the top left. Alternatively, I will be putting the link out shortly.

Brilliant.

I will be back in a second to greet any newcomers. But yeah, let us know where you're coming from in the chat. Looks like we've got lots of people joining from a variety of places. Ghana, South Africa, Bahrain, New York, Philippines. Wow. Fantastic. Right, I will see you in two seconds as we just wait for the last few people to join.

Foreign.

Thanks for joining today's Live training. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about a minute or so. We're just waiting so everyone has a chance to join, but in the meantime we'd love to hear from you. So let us know where you're joining from in the question box. And yeah, let me know what you want to get out of today's Live training. Just to note, we are going to be working on datacamp Workspace today, so if you don't have a Data Camp account just yet, make sure you register for your free account. I'm going to post a link in the chat now. Yeah, outside of that, let us know where you're joining from. We're also going to be having a workspace element, so it means you can code along with us today. So I will be posting a link to that. But for anyone that's already set up in datacamp Workspace, scan the QR code in the top left and yeah, you can, you can join along with the Challenges and Solution workspace. Okay. Hello everyone. Thanks for joining today's Live Training. My name is Reece and I'm going to be your moderator today. Uh, we're going to kick off today's session in about 30 seconds or so, but in the meantime, let us know where you're joining from in the chat and, yeah, let me know what you want to learn from today's session. Hello to Ramses Amal Nishant. Oh, that's me in the chat. We've got loads of people. Anna, Karima, Anayo, nice to see you all. For anyone new that's just joining, we're going to be working with datacamp Workspace today. So if you don't have a datacamp account yet, now is the time to register for your free account. I've put a link in the chat so you can create your free account. And I'll also be putting a link shortly to the challenge and solution workspace that will allow you to code along with us today. So, yeah, keep your eyes peeled for that. And we're going to get started any minute now. So I'm going to bring in today's host and yeah, that'll be Adele. Let's see, you know.

Hello.

Hello, everyone. How's it going? It's really exciting to be joined here by Matt Picard for today's live training on analyzing credit scores with Tidy models and R. My name is Adele. I'm a data evangelist at Data Camp. You know, there's a lot of exciting things that are going to be covered today. First of all, we're going to look at credit scoring data. Credit scores have a huge impact on people's lives from impacting whether someone can get a mortgage or. Or whether someone can get a business loan for that, you know, data science startup that they've been dreaming about since learning on Data Camp. Secondly, we're going to be using tidy models. This is a framework that has been brewing for years in the R community and is now at the point where it's the best choice for modeling in R. Is it better than Python scikit?

Learn.

Let us know in the chat. We'd love to learn from you. We'll be sharing some of your engagement in the chat here. I see quite a few folks from across the world. You know Vlad from Santa Monica. I'm going to show you here we have Gustavo from Italy, Ahmed from Turkey, Artur from Peru. So we have quite a few folks joining us from across the world. Do let us know if you prefer tidy models over SciKit. Learn. Thirdly, what we're going to cover today's session is dimensionality reduction using umap. Dimensional reduction is one of the most important techniques in machine learning, and it can really, really dramatically improve the performance of machine learning models, essentially delivering the same level of output with less data and less features. So I'm really keen to see how UMAP compares to more traditional model types like principal component analysis and onto today's instructor. So today's instructor is Matt Piccar, a professor at Northern Illinois University and the owner of the machine learning consultancy Picard Predictives. His work focuses on natural language processing, machine learning, statistical modeling and visualization. Matt is also the instructor for dimensionality reduction in our course on Data Camp. So we'll make sure to share that course as well in the chat. And now over to Yuma. I'll let you take it away.

All right. Thanks, Adele. Yeah, so I've been looking forward to this and it's really exciting to see people from all over the world joining in, and I hope that you find this helpful. Tidymodels is really kind of a really nice framework that, like Adele said, it's kind of been brewing in different packages and they. They've repackaged it in Tidymodels, and we're going to use that to build in some dimensionality reduction today as we analyze some credit scores. So I had a few slides that I wanted to show you, just introduce some concepts. So let's first talk about what dimensionality reduction is. We'll go and look at the first slide. Well, first let me tell you, we're going to be using a credit score classification data set from Kaggle. Why is that interesting? Because credit scores, like Adele said, have a lot of impact on business decisions and consumer decisions. But as a secondary goal, I wanted to use this context and learn about dimensionality reduction using UMAP and kind of just show you. I'm going to focus on using Tidymodels to kind of demonstrate this concept of information in data. Because when we reduce dimensionalities, we really want to simplify the data, but we want to keep as much information as possible. And I'm going to use Tidymodels to show you that. And the reason dimensionality reduction is important is we live in a world of complex data, big data, and sometimes we might be working with data sets that have hundreds of columns or features, and we're going to use columns, features, dimensions interchangeably. So let's, if we take a look at the next slide, the purpose of dimensionality reduction is just to reduce the number of columns. We're just going to reduce the complexity. And there's two approaches. The first is feature selection, and that's just simply Pretty much removing features that don't have any or very little information. And feature extraction is what UMAP falls under and it's actually combining columns, the information from columns to form new features. So let's give you a couple of illustrations about what feature selection, feature extraction looks like. I have some graphics on the next slide I want to show you. So this is basically a visual, visual explanation, feature selection. Imagine you have like a data set with six features and then you might discover that feature one and feature five really have no or little information. So for instance, they might contain just a single value and that value gets repeated. And if there's no variance in the value in the column, it really can't help us distinguish variance in the target variable. The next one is feature extraction and that's what we're going to be focusing on. So let's flip to the next slide. Feature extraction is about combining the information in two or more features into one feature. So we might be able to reduce it from these six dimensions to four dimensions by taking F1 and F2, an F5 and F6 and putting them in a feature together. We're just kind of grouping their information and that's what UMAP is going to be doing. So let's put, let's look at this chart on the next slide and I'll show you how UMAP kind of compares. So I won't go through all this, but UMAP kind of falls in the same family of feature extraction techniques as Principal Component Analysis tsne and here's just kind of a comparison between them. The main point I want to make on this chart here is one downfall. One I guess downfall of UMAP and T SNE is they become really difficult to interpret those features when they get combined. They're kind of like a black box, but they're really powerful. And we'll show you that actually in code PCA actually still lends itself to be interpreted. We could kind of interpret the new dimensions. That becomes more difficult with umap. So one more slide and then we're going to jump in to the code maybe two. So we're going to use Tidymodels and if you've used Scikit learned you're familiar with the whole paradigm. But essentially Tidymodels creates like this pipeline which they call a workflow that allows us to basically pipe pre processing code into the model fitting code into a post processing code. In Tidymodel speak we're going to do some pre processing, splitting the data, but then we're going to combine a recipe which is pre processing, like data cleaning with an actual model into one workflow. I think we might have one more slide. Oh, that's just the slide for the course. There's a link. If you want to learn more about Tidy models or dimensionality reduction, I'd refer you to that course. So let's jump into the code. Let me just get set up here. All right. Yep, we're up. Okay. Okay. So I'm going to go through the first part fairly quickly. We're going to set up our environment and basically when we're working in the Data Camp workspace, we're going to need to install the embed package. So I'm just go ahead and run that chunk of code there. And then we're going to load in the tidyverse libraries, the Tidymodel libraries. And the embed library is kind of a companion to the tidymodels package. It's kind of a supplement and it's what is going to provide us with umap. And then I'm just setting some options here just so that we can get some bigger graphs when we go plotting them. So let's jump down. I've got a URL to the data set here. And this data set is used in the dimensionality reduction in R course. And I'm just gonna go ahead and load that in. So I'll store it in credit df. I'm gonna use read CSV. And you may not see these. Oh, you do see these in real time. Okay, good. We had a few technical difficulties setting up, so I'm just gonna go ahead and read in data set and I'm going to do some basic cleaning here. So I want to. Our target variable in this data set is a credit score. And it's got three different levels. It's got poor, standard and good. And I just want to put those in the correct order so that our graphs end up making more sense. So I'm just going to use tidymodels mutate and I'm going to basically update that credit score column. I'm just going to use factor to take the factors out of credit score and reorder the levels. So that's all I'm doing here. So I'm going to order them as poor, Standard and good. And our whole objective with this data set is to be able to kind of distinguish poor, standard and good credit scores using our predictors. Then let's just go ahead and take a glimpse of our data set so we kind of know what we're dealing with on that chunk. There we go. So if we scroll down, you'll see we've got things like month, age, occupation, annual income, go on down. And there's a few more that are a little more complicated, like credit utilization ratio. That's basically just the percentage of their available credit that they're using. And what we want to do just to become familiar with this data, we essentially have, we have 23 columns, so that means we have 23 dimensions. But our, our target variable is our credit score. That's the very last one that you see in that glimpse. And so we really, we have 22 predictors and we're going to essentially use UMAP to reduce those 22 predictors to two and try to maintain as much information as possible possible. So let's go down to the next section and we're going to do some exploration. And in this exploration, basically our objective here is just to start playing around in the data. And it's almost like we want to say, okay, which of our predictors are going to do a good job separating the credit scores? We want to be able to differentiate between the credit scores. So, and I'm just going to run the code here and I think it's already run. So let's take a look at that first graph here. We have plotted annual income and we've just done a density plot with those. And you can see the axes are a little small, but along the x axis is annual income and you can see the distribution of those. The red is poor credit scores. And so people with lower annual income tend to, tend to have poor credit and people with higher annual income. If you take a look at that blue line, it's kind of higher on the right hand side. So there tend to be more people with good credit that have a higher annual income. So what we could say is in the dimensionality reduction standpoint, we could say that annual income has good information about our target variable, credit score. So let's just kind of look at another plot here. If we scroll down, we're going to see, we're going to do a density plot of age. And so again you can see that it separates our different levels of credit scores. The red line are people with poor credit scores. So people that are to the left are people who are younger. It's hard to get the whole plot in here. And then the blue line has that hump on the right. So basically people who are older, more mature, maybe more life experience, tend to have better credit scores. And so again, we could say that age has some information that's useful for determining credit score. And let's just go take a look at a couple other examples. I wanted to do a scatter plot here. And basically just because that's how we're going to visualize the effects of umap. So I kind of wanted to get into that mode here. We applauded delay from due date, which the delay from due date is just the average number of days they're laid on a payment. So that's along the y axis. So things going to the further to the right, they would be more late on their payments. And then along the. Sorry, that's along the x axis and along the Y axis is credit history months. And that's just the number, the length of their credit history, how long they've been like recording credit history for this person. And what we see here, if you see that kind of blue square in the upper left, there's kind of a dense blue square. So that would be our good credit people. And so basically what we can conclude from that is people who have longer credit history and they tend to pay on time more often, tend to be good credit, have good credit scores. And so what we're kind of looking for is just, okay, where do we see the blue, where do we see the green Standard scores? Where do we see the poor? And you can see that credit history months and, and delay from due date tend to have some information that help us identify people with good credit scores. So again, we're just going along and we're going to jump down and do one more. We're just trying to get a feel for which of these predictors might help us separate poor standard and good credit scores. And so the last graph we have here, we've plotted num the number of credit cards a person has along the x axis and the credit utilization ratio along sorry, along the Y axis is number of credit cards. Along the x axis is the credit utilization ratio. And that's basically what percentage of what percentage of the available credit that they have are they actually using. And if we just scan left to right, you don't see any patterns where you see more red or green or blue dots on the left or on the right. But if we go up, which is the number of credit cards they have, you can see that the top has poor red dots more often than the bottom does and the bottom tends to have more blue dots. And so number of credit cards seem to differentiate whether people have good standard or poor credit scores. So again, just to wrap up, all we're trying to do is just Kind of explore and find things that differentiate. We want to find the things that have information about credit score without actually knowing the credit score. And then we can like say, enter umap. Because we could spend all day looking at different combinations of these predictors and plotting them and it would take us a long time to, to visit all combinations. But if we enter, if we bring in umap, UMAP is an algorithm that allows us to basically reduce those 22 predictors into a smaller set of dimensions and we're going to reduce it to two. So let's show you how that works. The way we're going to do it is we're going to leverage the Tidy models framework to actually apply UMAP to a data set, to our data set. And we're going to do that inside a recipe. So remember, a recipe is just a place where we can put a bunch of pre processing data. Think of it just like a cooking recipe. It's a series of steps that we could repeat on a different data set. Let's take a look here. I'm going to scroll down to our first code block here and to create a recipe, the first thing we're going to do is create a recipe object and then we're going to add steps to it. So let me show you what that looks like. So we'll call it umaprecipe. And here's our recipe object. And inside our recipe object we're going to give it a formula, kind of like a regression formula, just to be able to define that our target variable is credit score. And then we want to use, we'll use the period to indicate that all other predictor variables should be used to try to predict credit score. This just gives the recipe a context for how to treat the steps. And I'll show you why that's important here in a minute. Then we're going to pass it our credit df. And that code right there creates our recipe object and then we're going to use our pipe. Then once we've created an object, we can start adding steps to it. Just like in a recipe, you have different steps that you have to take with the ingredients. This first line of code, we've said this is what we're trying to make. That's our formula. And we said these are the materials we have, that's our data. And now you need to go through these steps. And our first thing that we need to do when we do, when we use umap is we need to normalize the data. The reason we normalize the data is just so that all the predictors kind of are on equal playing ground so that some predictors who have a larger range, like salary that might go from like 20,000 to 300, 400,000, don't overpower the UMAP algorithm. And we're going to apply the normalization step to all our numeric predictors. And, And Tidymodels has the these really useful, helpful, fun helper functions so that we can specify kind of in plain English, these are the, these are the variables I want you to apply this step to. And so that's going with a few things to note about this. We're not going to normalize our target variable. One, it's categorical, so it's not numeric. And two, it's a target variable and we don't want to change that. And that's what that all numeric does. And then our next step, we're going to pipe that into our step view map, which comes from our embed package. And we want to apply UMAP dimensionality reduction on all our predictors. So implied in that is that UMAP can handle categorical variables. So we have some categorical variables in our data. So we're going to apply it to all predictors and we're going to tell it what our outcome variable is, our target variable. And we'll use vars to tell it. That's the variable in our data set. We're just telling it right here that, look, credit score is our target variable. And the reason that's important is because we want UMAP to be able to say, okay, which of these 22 predictor variables contain information that is useful in separating out the three levels of credit score. So we're just kind of giving it a heads up there. So in all this code right here that's just, just like Scikit Learn, it's just going to instantiate those objects. We haven't trained anything at this point. So I'm going to run that. Everything looks good now. If we jump down to the next code block, what we want to do then is we want to actually fit our recipe to the data. So it's kind of like training it. And we're going to store that in a variable called umap. We're going to eventually extract the data that's fit. And so we're going to store it in UMAP credit df. We're going to take our UMAP recipe from above that we just created and then using the pipe we're going to call prep. So Scikit Learn, we'd call fit. Since it's a recipe, they chose cleverly to call it prep. We're preparing the recipe, and what that essentially does is that that fits the recipe to the data. But because we want to actually visualize it, umap is mostly used for visualization. And there's a link down here. You see that? Here are some nice examples. Link just below our next code block. You may want to click on that and take a look at some of those. There's some pretty cool examples of some visualizations. Just for fun, I'd encourage you afterwards to take a look at some of those visualizations. Really, we aren't using UMAP to its fullest because umap is very useful when you have, like hundreds of dimensions and we've really only got 22. But we're just doing a little toy example here. Let's go back to the code and see. We prepped our recipe, but in order to get our data out that's actually been fit and transformed by that recipe, we're going to use a fun little function called juice. I applaud the Heidi models folks for their creativity. It took me a while to figure out guice. We're just trying to squeeze the data out of the recipe. When we run this, it's going to return a data frame. All right, let's see. We gotta go debug a little bit. One sec. And so up above, we're gonna look at some of the. Oh, I know what we need to do. I forgot one parameter here. If we go back up above our code block where we created the UMAP recipe. Keep going. It spits out a whole lot of stuff. Right up, up right there. One up right there. Okay, so one thing I forgot here is umap. We need to tell it how many components or dimensions we want in the end. And so we're gonna do number num. Components is equal to two. So that's a pretty important step. We could make this three. We could make it four. It gets harder to visualize beyond two or three dimensions, and that's why dimensionality reduction is often helpful. So I'm going to run that one more time. I'm going to come down to the next code block and I'm going to run that code. Why are we. Let me see what I'm missing here. The solution ran earlier today. Oh, I know why. Okay, I forgot an S on step normalize. It needs to be all numeric predictors. So let's fix that. There we go. Now I'm running the prepare code block again. Let's wait for it to finish. UMAP tends to be more computationally efficient than T sne and so that's one advantage of it. It's still running. You'll see it update in just a minute. There we are. All right, I got through that bug. Now, like I said, the GUICE returned the data after UMAP was applied. So I'm just going to take a glimpse of that data so we can see the result of that. So show you here. There we are. There. I'm just going to glimpse it. And notice that we now have two dimensions. Those 22 predictors have been embedded into two dimensions with UMAP. So to actually see what that looks like and more importantly, to see how much information UMAP was able to retain from all these 22 predictors, we're going to kind of test that by just seeing how well you. Those two UMAP dimensions separate our three credit score levels. So we're going to plot. We're going to take that UMAP credit df. So I see, Alejandro, that's a great question. Which, what are the pros and cons between PCA and umap? Let's go ahead and look at this in this ggplot and then I'm going to open up for questions and I'll answer that one first. So we're going to pipe data into ggplot. And we're going to say along the x axis, take our UMAP dimension 1 along the y axis, plot our UMAP dimension 2. Again, I guess we just saw those up above and from the glimpse, and then we're going to make sure that the data points are colored using the credit score so that we can see them. And we'll go ahead and add on a GEOM point. So we're going to do a scatter plot, and I'm just going to set the alpha parameter to like 0.2 to make the data points translucent so we can kind of see through them if they're stacked on top of each other. There we go. So now you can see based on those two dimensions, it does a pretty good job of separating our credit scores. We can see some clear green blobs. We can see some clear blue blobs. And towards the bottom, you see kind of a red blob. That shows our poor credit score consumers. So to, I believe it was Alejandro's question, what are the pros and cons between PCA and umap? PCA is going to be interpretable, which is kind of a big deal. You know, there's this big push for interpretable, interpretable AI. So let me just explain that if we scroll up to where we saw our UMAP dimensions in our glimpse, like we have, we those are essentially coordinates for each of our plots and it's not trivial to interpret those new dimensions. With PCA we have things called loadings. So we can see how the original features, original variables loaded onto the new factors or components. In umap, it's not as easy to be able to say, oh, UMAP Dimension 1 is composed of this variable and that variable and that variable to this weight. That, that was one advantage of pca. So if you need interpretability, you're going to want to go with PCA and the recipes package and tidymodels actually has a step pca. So you could basically replace that in our recipe, replace the step UMAP with step pca. UMAP also has the advantage over PCA that they call it global and local interpret interpretability. So if we scroll down to the plot again, basically what that means is that like that green blob, the distances in the points inside that group, they have meaning. So the ones on the outer edge tend to be like, less like the ones on the inner of that green blob. And then the distance from the green blob representing standard credit to the blue blob to the upper right of it, that has meaning. And so the problem is we don't know what that meaning is because UMAP1 and UMAP2 are kind of black boxes to us. Those dimensions are black boxes to us, but it's interpretable from a distance standpoint. So they call that global and local interpretability. It has that advantage over T sne and I'm not going to get this right. I'd have to go back and look at the slides. But I believe T SNE has local interpretability, but it doesn't have global. So that means that inside a certain group, I, you can infer things based on the distances from the members of that group, but you can't infer things from how far that group is from another group or another cluster. And so UMAP kind of brings back some of this. Oh, I can make some decisions or inferences about how far this group is from that group. So those are some of the main pros and cons between UMAP and its other siblings in the dimensionality reduction. So at this point I wanted to, wanted to open it up for other questions and we'll take about five minutes for questions and then we'll jump back into the code. Let me just look Ah, yes. Thanks, Mohammed. How to relate UMAP1 and UMAP2 to the original features for interpretation. To be honest, I'm still trying to figure out how to do that in tidy models and I'm not sure it's possible to do in UMAP because of the way the UMAP algorithm works. And so that's something I'm still looking into. To my knowledge, I have not been able to find kind of like the factor loadings type of interpretation that you would be able to get with pca. And so it's definitely more of a black box than pca. I think that question is how should we decide when we should use PCA and when we should use umap? I would say if you're just trying to do kind of predictive analytics and all you're concerned about is we got to predict this accurately, I would probably use umap because you don't need to know why it's predicting it that way. If you need to be able to explain like to a manager or to your boss, this is why it's predicting or categorizing this data point like this, then you're going to want to use PCA because you're going to be able to see like how what makes up the new dimensions in principal component analysis. And Principal component analysis, we call them components, but they're dimensions, they're new features. And so if you need interpretability, use pca. If you need just a powerful algorithm to reduce a whole lot of dimensions into very few, I would say use umap. And when your concern is only to predict accurately. So that kind of leads us into the next portion of our code. So let me explain this. Going into this, we're gonna build two models. We're gonna build just a bare bones model. It's gonna be a decision tree to try to predict credit score based on our 22 features, our 22 predictors. And then the reason why we're gonna build that and then compare it to a model where we've implemented a step UMAP in the recipe is we just want to get a feel for how much information we lose in that reduction that UMAP does. So our first step is we're just going to create a model, a decision tree model. We're just, we're still going to put all the parts into a tidy model workflow, but essentially the recipe is going to be empty and we're just going to have a model. So let's go down to this next code block and we're just going to do a little Bit preparation because, because we're going to split the data into train and test data sets and that is based on randomness. We're just going to set our seed for reproducibility and Tidymodels. One of the packages inside Tidymodels, I believe it's the R sample, has this function called initial split and that's just essentially going to configure our what would go into our training data set and what would go into our testing data set. So we're going to call initial split and pass it our data set. Then down here we're actually going to extract our training data set using the training function. And I stored the split into credit split. So I'm going to pass it the split object. And I have to say that I did for a long time kind of salivate, you know, being mainly an R user, I would look at Scikit Learn, I'd be like, ah, that, that is so easy to create training and test sets. And it wasn't that easy in R. And so I'm really grateful that they introduced this new interface that makes it pretty easy. So then we're just going to call testing to pull out our test data set. So we've got our train data set and our test data set. I'm going to go ahead and run that. We're good. So now we split our data. Now we're ready to model. So let's go down and we're going to build a workflow without UMAP first. So I'll step you through each of these parts and then when we build it with the umap, we're just going to kind of clump the process together. The first thing we want to do is create a recipe. And since we aren't applying umap, we're just going to create an empty recipe object. So here we're basically saying, look, we're trying to predict credit score and we're going to train the model. We're going to process the data using, process the recipe using the train data, which essentially ends up being we're going to train the model using the train data. And you might think, you might ask, you know, why are we storing the formula in the recipe? Well, like we saw before, if we can give it the formula, then it can interpret things like only apply this step to all numeric predictors because we've just told it in that formula right there what predictors are and what, what the target variable is. So now let's go down here. And now we're gonna create a model Object and Tidymodels is really cool because it's abstracted away all the details of different models. So it's really easy to swap in different models in our workflow. You can see here, very generically we call decision tree. Decision tree is an interface to a lot of lower level models. We might be able to create a CART decision tree, an ID3 decision tree, and we could change that just by specifying some parameters here in the decision tree function. But we're going to accept all of the defaults for this demonstration, except we need to specify. Look, we're trying to create a decision tree with the goal of classifying data as opposed to creating a regression decision tree model. And so that's why we do that. I'm going to go back up and run our recipe and run our decision tree model. So essentially that just instantiated empty recipes and models, it configured them. And then when we get down here, we're just going to create a workflow model stored in DT workflow DT meaning decision tree. I create an empty workflow with my pipe. I'm going to call Add recipe and pretty intuitively I'm just adding this recipe to my workflow and then I'm going to do the same with my model. I'm going to take that model that I created up above and add it to my workflow. Again, I haven't fit anything at this point. I have just created the objects necessary so that I can actually pipe my train data through and do the fitting and such. And so that's what we're going to do here. We're going to go take, we're going to store the fitted model in dtfit. And so we're going to take our workflow. And we're going to use the pipes to run it and say please fit it on the train data. So this is very similar to scikit Learn if you're familiar with that. So the fit is just essentially going to fit our model and it actually calls prep on the recipe too, which in this case doesn't do anything, but it will in just a minute when we add steps to it. So that took a little longer to run and now we've actually got a fitted model in there and we can use that model to predict our test set. So we'll store our predictions in a data frame and we'll take our test set and we're going to pipe it into. Essentially a predict function. But I'm going to use bind columns, actually take our predictions and bind it to our test set. So I'm gonna create a new column called Predict and this predict function right here, that comes from Tidymodels, it's very similar. Scikit Learn where we're going to basically say we want you to take this fitted model or workflow and pass in our test data and give us the, the predictions. So this is going to generate predictions for us which we're going to assign to a predict a column called Predict. Then we're going to bind that to our, our existing test data. So that test data will have our 22 predictors and our target variable. We're just going to add a, a predict column to it. So now we've got our predict data set. I should call this something different, not dt, but df where there we've got our predict data frame and then Tidymodels, it's like an ecosystem of packages. It has a nice set of packages that allows us to perform some measurements on our results. So there's this one called F measure, which is the F measure which combines recall and precision measures just so that we can get a feel for how our model performed without dimensionality reduction. So I'm going to pass it. The predicted scores, my credit score target variable. And then there is our predict class that's going, we're going to use to compare the actuals to the predicted. And that's just a, that's just a column in our data set in our predict df. So the takeaway here is that we have an F score, an F measure of about 62.15%. That's our output here. So that's going to be our benchmark. Now let's turn around and let's actually repeat all this. But we're now going to repeat the same steps. But in our UMAP recipe, we're going to actually add UMAP steps to the recipe. So here we go. So here's our recipe and we instantiate recipe object credit score is our target variable and we're going to use all others as predictors and we'll use, we'll pass it our train data. Now this is different from the last time where we had an empty model. We're going to do step normalize just so that all the predictors are on equal playing ground. Then we're going to add another step, our step view map. We'll apply that to all our numeric figures. We need to tell it what our outcome variable is. So it can kind of do this analysis of which of these variables contain information about, about our credit score. And then we'll say just give us two components or two dimensions back. Then our next step is we want to actually create the decision tree model. So we'll instantiate that using our high level decision tree function. Look, we're doing a classification model and then down here we're going to create a workflow model or workflow object and we're going to add recipe and our model to that workflow. Here we pass it our DP recipe which we created above. Then we're going to add. Model DT model which we created above down. Well, just like above, we're gonna fit our model. So we take our DT workflow object and we call fit on it using our training. And then let's predict. Remember, at this point the difference from above is, is basically this. We've added our, our UMAP pre processing code. So if we come back down here, we're going to create a data frame of our predictions using our test set, Bind them to our test set, have a new column where we store our predictions. And predict the test set there. And now this is where the drum roll is. We want to see how well our UMAP reduced implementation of this model compares to our base full featured model. All right, hopefully got all the code written correctly. All numeric predictors doesn't exist. Oh, I need some parentheses there. All right, give it a minute to run. It's going to train the model and give us an F measure out. And while it does that, I will say, you know, whether the F measure comes in a little lower or a lot lower depends on our context. So you can see we got roughly 54.5 for our F measure, whereas up above it was like 62.2. And so what I want to use that for is just to kind of say, look, we lost a little information by reducing it to two dimensions. But given the fact that we went from 22 predictors to two and our model still performs a little bit better than chance, well, that we can't interpret that as chance, but we, we dropped a little bit in our F measure score. That may be okay. It just depends on our context. If, and this kind of goes back to answering one of the prior questions. If speed is important to us, then UMAP might be a good solution because it's gonna reduce our data. So then when we actually feed predictors into our model, it's dealing with less data and we might get a little bit performance gain. But you all, you also have to remember that we're still applying UMAP to any data that we feed through our trained model. So there's some computation there. If accuracy is of utmost importance, then and we're like we can't take that hit in our model, then we probably want to like maybe use the original features. But again, the purpose of this was just like we did up in the plot. We saw the information that UMAP was able to take and in two dimension it was, it was able to separate those three credit score levels. And we just did this same exercise kind of in a model building context. So I mean you, you decide going from a 62.2 F measured to a 54.5, is that a significant hit? But the point is it kept a good amount of our information. So that's all for the coding part. Why don't we open it up? We got a few more minutes. Open it up to questions. What questions can we answer? That's a good question. Is it Junaid? What are some of the best practices? Well, I'm going to answer a different question. I will just point out that UMAP is significantly more computationally friendly than T sne. So I would say if you're concerned about performance, it's and memory usage that UMAP would be the way to go compared to tsne. Other than that. Oh, the one thing that, the one thing that I should mention and if you go look and Google like embeds step umap, you'll see a bunch of hyper parameters that you can use to actually get a better fitting model. We accepted all the defaults for umap, but you can actually tune different hyperparameters and depending how many dimensions you have in your embedding and whatnot, that's going to affect memory usage and performance as well. For instance, if we using the parameters that we've seen, if we decided we wanted three components instead of two, we would expect that our end decision tree model might perform a little better because hopefully we're not removing as much information when we reduce down to three dimensions compared to two. And so you can tweak that. But there's a bunch of other parameters that are really beyond the scope of our discussion today. I would just encourage you to go look at the step UMAP and look at all the hyper parameters you can train. So thanks for asking that. Any other questions? Why do we call decision tree as a lower model? I'm not sure what you mean by a lower model. Can you expound on that in the chat?

I can think I could add some context here.

Matt.

I think we met as a baseline model. Why do we use it as a baseline model?

To a certain extent, yeah. Like, why did we choose that compared to like random forests? I was really just looking for a clean model algorithm that we could interpret pretty easily without like the extra confusion of like, random forests. So just using that as a comparison, random forest includes a bunch of other randomness that would be hard to kind of see through. But also random forests wouldn't be very helpful because we only have two dimensions after umap. So I, I just kind of, I, I picked decision tree just because I thought it would be a good illustration to show the information loss with umap. Is that the answer you were looking for?

Definitely think so. So we still have one more question before we wrap up today. I have a question here from Carlos.

How do you see adding these two new dimensions as features to our model with all the rest of the variables? So we usually wouldn't. If you mean like adding them back to the data set, that would. Essentially what you'd be doing is if you added the UMAP dimensions back to your data set, you would be duplicating information because all the information in your existing. All the information in your UMAP1 and UMAP2 would already have come from your existing variables that you started with. So it's about compacting information into two dimensions, if that makes sense.

That definitely makes sense. You don't want to add duplicate information within your data set. Now, one thing I'm going to echo, I'm going to add a comment here, not a question, which I think echoes a lot of the sentiment of everyone here is from Junaid. Thank you so much, Matt, for the detailed explanation of UMAP dimensional production and credit risk scoring. This is a very, very informative and highly educational and entertaining live training. I wanted to say, because we are running out of time, a huge thank you to Matt, everyone that has joined us today. And make sure everyone to check out Matt's course on dimension dimensionality reduction in R. Matt, any final words before we wrap up today?

Yeah, I just. Thank you for the opportunity. This has been fun. I've learned a lot of things in the process. I tried to make it a little bit more conceptual related so you wouldn't necessarily do this in real production code, but it was to help you understand the concepts and that was the same approach I took in the dimensionality reduction for our course. So thank you, Adele.

That is awesome. Thank you so much, Matt, for joining us today and thank you all so much for joining today's live stream. See you on the next one.