Biological Data Insight & Modeling

All right, so biological data insight and modeling is the theme. There's the abstract. Data modeler has been hanging around for quite a few years doing a lot of industrial design analysis, some sports teams, other stuff. But in the last few have gotten into the biological space, it turns out that omics is the theme maybe. So you got proteomics, genomics, lipidomics, metabolomics and it's just cranking up the number of variables that you're looking at. Still in the medical space they're not having a lot of patients. So now we're getting to these sort of super fat data sets we have. We're in, engaged in a few NSF studies, doing analysis but today we'll be looking at data from bone regeneration because in terms of bone healing you want to be able to detect early on that you need to have an intervention if you're not progressing properly. This particular data record reasonable immune response. This is metabolite data. This one's pretty tough. 8,000 variables and only 20 patients. We're trying to identify the biomarkers that are going to be indicators of the immune response. Cancer vaccines a near and dear one to my wife's heart because her sister died of glioblastoma brain cancer. She's done quite a bit of work of this. Basically what the cancer vaccine approach is you've got a, it's car t therapies. So you want to train the immune system to recognize the cancer cell. Because we have mutations all the time in our bodies, the immune system detects it cleans them out. Some of the, you know, you get cancer when the immune system says okay, you're friendly and they can run sort of amok. Now the ideas for the car t therapies is that you want to train it, but the problem right now is growing the cancer cells doesn't scale well. It's very expensive. Millions of dollars to treat one patient that's not hitting the overall need for cancer. Then single cell multinomics is just basically you can take the same cell measure over and over again. So you wind up getting a huge data set but a limited number of patients and it's noisy. So you have to do special things in that space. I am not going to do anything with the data model or GUI other than to say that it is a thing of wonder. I went for many years as being a command line notebook bigot and I was forced into getting the GUI built. But the reality is when you're dealing with this sort of data sets, you're very interactive. You're taking your data and focusing, you got a huge haystack. You're just trying to make that haystack smaller and smaller and figure out what really matters. And it becomes an interactive thing of doing a thing. What if I want to go look at the data and you're just bouncing back and forth and having something that's file driven and, and supporting that workflow is a good thing. So we've got this data. So what do we do? Just before we get into this is sort of general to data modeler. If you've got big data sets, you have ordinal gp basically subsetting the data, figuring out what's going on. Because you can evaluate a model as a good or a crappy model on a subset of the data, you don't need infinite precision. In the initial stages of this sort of evolutionary process of modeling missing data. Very common approach that we do is because what we're going to do is we're going to look at, we're going to develop algebraic models on explicit variables from the data. So, so unlike a neural net which assumes every variable matters, we're assuming a handful matter and we're going to throw in a constraint that the variables that we use have to cover the data set to a reasonable 75%, 90%, whatever you specify. But this is how we're able to handle missing data. So we're not doing an imputation stage model support. If you've got, if you've got a handful of records, you don't allow 27 variables. Right. So we have a requirement that no more than 20% of the defaults you can have the number of variables, the number of basis sets, sort of top level additive terms can be no more than 20% of the number of data records. And you can sometimes you want to specify that a little lower. Just be cautious. We have the issue when you start dealing with small data sets that. We just have quantized data, so we've got interstitial regions in our parameter space. How do we handle that? Well, one of the approaches is to do what I call robust models. It's just basically using interval arithmetic and guaranteeing that I don't have a singularity in between 1 and 2 and 5. So we have this issue. The other is that you can analyze the models with anova trim clean out terms. Models that don't at the top level pass an ANOVA test. So the statisticians can be happy with you eventually because we're going to go through and we're going to do this modeling and it's going to be stochastic and we're going to have thousands and thousands of models and a handful of variables that we look good. We're going to build ensembles, models, and these are give us trustable models. Okay, all right. So data characteristics in sort of the case I'm looking at, you know, very few records, tons of variables. It's ill conditioned with correlated variables. Just as well as having too many missing data, we got to only have the ability to analyze the data because people love if they don't have data to substitute in a zero. A zero is not necessarily a zero, right? Sometimes it is, but oftentimes it is. We need to handle that and if appropriate, convert it to being missing and then quantized is a big issue when you only have a little bit of data. So what this is here is a snapshot of metabolite data. So we've got, you know, 8,000 variables. And if I look at the models that are developed, I mean this is not something anybody, any statistician would propose as a model. Right. But the evolutionary process identified these things and they're passing. I can do an ANOVA test on it and it's legal and it's accurate and it's just what this build backs. But what we're trying to do is in the early going of this is, is focus things down and figure out which variables matter. And occasionally we, once we figure out which variables matter, we figure out which ones are the most controllable. But our whole assumption in this process is just really what makes a good model. It has, it should be simple, right? We're assuming a handful of variables, you know, want to be a small compact model and it want to be accurate. And that is the evolutionary pressure that we're applying. And in this case we have 1000 models that we did out of the initial search. So if I look at those thousand models, you know, first, one of the questions I want to ask is how many, how many variables do I need? So someplace two variables gets me to better than a 90% R squared. I mean that's for a biological system. That's going to be pretty attractive to me. Right? Don't chase R squared I guess is my theme in this. But what we're doing here is we're explicitly exploring that complexity accuracy trade off. And because of that, because of the ability to do complexity control, we can do things like build ensembles and have trustable models. So you know, I guess the other thing here is that is what you notice now this particular variable combination map is Showing that for all of the models in the data set with two variables, what are the combinations of variables that are there? We see very strong presence of TNFA. I can't read it, but it's 93% of the bottles of that set. So we're, you know, we've got combinations. We can start to see sort of dominant variables for doing sorting. All right, so we're starting to get a feel for about how many variables we need and say what variables matter. Now the. We can really basically do three things. So we've got 1000 models and we look at our population models and we say, well, which variables are most prevalent? So what's variable presence? Because presumably if they're in a reasonable fraction of the good models, then they're useful variables and they've percolated up to the top. We can look at combinations of variables and say, you know, this particular group of three variables, you know, produces good models, and that's a useful thing. Maybe I want to go do some modeling with just those three variables. And then we can look at associations, which variables play nice with one another. Oh, geez, sorry. So looking at variable presence. So what this is again continuing the immune system response with the metabolites and all lot of variables went in. Only a couple hundred were even in the interesting models that we selected to do it. Now the thing is, it is a needle in a haystack problem. We've got of possible triplets, 10 to the 11th possible triplets. But over the course of my modeling of only done millions of models that have been explored. So what I'm gambling is that a variable that's come through in this stage is something that is interesting. I want to do a lot of modeling before I make a strong conclusion and, and let things happen and be sort of slow as I try to make this haystack smaller and smaller. One of the other neat things on this is, is this, you know, this variable right here was in over 10% of my model. So this is, this is a variable that's. That's pretty cool. 5, 098. If I, if I look at just strictly linear correlation, See how good my fine motor control Is, it's in 3.7%. You know, you know, just strictly from a linear standpoint, it's not something that would pass threshold if you were doing an initial cut based on, on correlation. Okay. So no, it's, it's a score that they have for functional effectiveness. I don't, I don't actually know how it's. It's a this case, it's a continuous variable. And then variable combinations assert obviously that if it, if it works, maybe it's worth doing. You know, you get into situations where I'm looking at variable selection and you know this, they're all metabolites for this particular situation. But you can have variables that are controllable, variables that are observable. Let's, let's focus in on, on the ones that are controllable. Okay. Associations is the other thing that we care about and that's just basically which variables like to belong with other variables. So I've gone through and this is sort of an association plot. If I look at this sort of dominant triplet, we see that it's not really highly linearly correlated. We can look at the, at the coverage of the design space and they tend to, observation is, you know, that tends to cover the design space better. So there's sort of a, maybe an analogy to mutual information that's, that's an evolved property. Meta variables are sort of building blocks that can sometimes provide insight into, into mechanisms. Ensembles are what really is the neatest thing out of this game. So we've got a bunch of models, we've now down to three variables that we're going to use or two variables that we're going to use in our ensemble. And we've gone and done some modeling and now we've still got 1,000 models that we can choose from. How do we choose the model? The problem with the model is the model wanders off to Never Never Land and you have no clue of what's going on. If you build an ensemble, if you build an ensemble, the ensemble has a special property that it agrees where there's data. Otherwise they wouldn't be good models. But we're going to choose them based on their diversity. So if we ask them to extrapolate if they go into new regions of parameter space or the system goes through a different change, then I could detect that. Okay, so this is really neat stuff. Now it assumes of course in any data driven modeling that your actually data covers the design space and a variable that's real important was not held constant through the whole thing. But it gets around the issue of the reason we can handle these 20 data records is we don't need to partition it to test training and validations because we've got two things. We've got explicit complexity control and, and we can use the power of the crowd to give us an overall behavior. So we get that trust metric. So this is just some bone regeneration. So this is problem of if the military's funding this one just because of war injuries. So you get people fractures and such things. So you're trying to get the bones, man. The idea is that can you early on predict what's going on and do necessary interventions? But if we're going to take the metabolites out of the patient, which ones are triggers that's progressing well or not progressing well, or indicators rather, here there's four different ensembles and different combinations of variables. From my perspective, they're all roughly the same. So if I can chase R squared and these four variable ensembles get a little bit better. But from a practical standpoint, the ensemble performance doesn't really differ a whole lot. But what we see here, if you look leptin at week nine, if we hold the other variables constant, where those green dots are, that's the reference point for this graphic. If we move that without changing anything else, we're going into an unknown region of parameter space. And so we, our trust metric starts, expands. Okay, so this is, this is pretty cool stuff. Data cubes and summary statistics. So this is the single cell RNA where they, where they've got, let's see, you know, 8,000, 8,000 experiments, but they only have 20 patients. They basically have the same data record over and over again. So we can model on the metabolites that they measured out of this or the features that they measured the RNA actually out of the single cell. Or we could do summary statistics on each group of patients and maybe that tells us more that does the spread, does the interquartile range of a particular RNA matter or does, you know, so we can actually get more insightful models. But this one's sort of neat because the ones up, you know that there's a huge gap in the, in the response region. And this, this is the problem with dealing with, with small data sets. So there's interstitial gaps and then trade off analysis. If I, if I have. This is the case getting back to the cancer cell stuff where you've got total live CD4 and total live CD8 cells. The CD4, if memory serves, if I'm correct, is controlling the immune responses. How aggressive your body is in terms of reacting to the, to the cancer. The CD8 is a modulatory T cell. So it's, it damps things down because you, so you don't get a systemic reaction as you know, and kill yourself in the process of curing yourself. Okay, so. All right, so conclusion is, you know, what we're creating out of this process is a white box model. It's interpretable. We can build these ensembles which is going to give us the trustability factor. The humans in the loop. I mean the assumption is that a handful of variables matter. But I am responsible for deciding which handful makes sense. Which does this agree with my mechanistic understanding of biology. It's essentially nonlinear models. But this is an issue here because everybody knows biology is linear, right? But the other thing is we're not doing hyper parameter tuning. We're not tweaking out the yin yang to get an answer. Was just sort of falling out for us. So very unique capability for this class of data. It's a good capability for, for all classes of data. That sort of spreadsheet style Data sets of 10, 20,000 variables and less than a million records. But for this particular super wide stuff that we're running into in the biological space, it's really the only game in town because partial least squares get you maybe to 50, 100 variables. Lassa doesn't do anything for you obviously don't have enough data records to do, you know, almost anything else. You've got to really isolate down to the factors that matter. So that wraps it up and thank you for not being out on that bus that's sitting there waiting for you. And if anybody's got a question, I'm here.

Is it? So I assume I came later, but any kind of dimension reduction you do Normally transpose this 20 by 10,000 matrix, am I correct? You have like super wide table, right? We've say 10,000 variables and 20 rows.

Right.

So if I want to do any dimension.

No, you do it directly. You do it directly because the idea is that we're going to do. What we're going to do is we're going to say that we're going to allow. We don't know which ones, but we're going to allow three variables into our models. So you're doing a dimension reduction 2, 3 variables.

Explicitly I was talking more like making decomposition type of algebraic. Okay, right. My point is what Imagine I do thin singular vertical composition on the transpose matrix. Then I can prove.

Right? So partial least squares. I mean that's, that's a partial, partial least squares regression is, is essentially what you're talking about. The decomposition. You, you look at the dominant singular value and you say, you know which terms are dominant in the dominant singular singular value that contains, you know, and how far deep do I go into the energy gives you a linear model. That's a good thing if you know. So here Here we're not constraining to linearity. And if you look though, you know, the trade off curves are, they're not linear. Right. I mean this is, you know, these, these are biological systems. They're not linear. So to impose a linearity constraint isn't from my perspective valid. Now it does work. I mean the whole point of linear modeling is it works in engineering solutions. It's great because we can make approximations there. So. Okay, I have been told to sit down, shut up, but I'll still stand here if anybody cares about it. Otherwise, anybody who's not interested, just. Yeah.

Another kind of question. If I discretize this and then instead of looking for correlation, the variables themselves are moving to association rules between the discretized.

Right. So maybe you can do a mutual information thing that's, that's very computational heavy. So, so it's, I mean, you just, I mean the scale of the problem is such that a mutual information approach really doesn't scale into it.

Okay. Anton asked this question and cluster them. So instead of 10,000 variables, he has 10 clusters. And then he was trying to use each of the clusters to try to predict response.

Right, right. So I mean there's obviously you get highly correlated variables there. They're highly correlated variables.

Makes, you know, ANOVA meanings. You know, as soon as you let in the first correlated variable, the other seven correlated variables look like garbage.

Right, right. So, so what happens sort of neat out of this is that the models that are developed are, you know, they'll pass. Even though they weren't told to. They'll pass ANOVA test. Okay, because they're, because, because it's basically there's a focusing in, on, on combinations, you know, so, so if, if two variables are highly correlated, the energy basically gets consolidated into the ones that are most predictive. The higher information value rather than the higher linear correlation. Okay, Bob, I ignore it. In this case I don't have any missing data, but in general I just ignore it and don't feel. You don't have to say, listen to me. Well, so remember you're dealing with explicit variables. Right? So, so what I'm, what I'm going to do is each model, when it's evaluated, I'm going to look at the variables used by that model and I'm going to say what fraction of those, the inputs of those variables for that model are completely numeric. And if they're over 75% or over 80% or over 90%, then I'm going to say, okay, I got enough support that I can despite the missing data that I can if it's a, if it's a high quality model in terms of, of its R squared so subset of variables. Right to the available information Next model, different set of. Right. Right. So. So I mean in this fashion you can handle where they're scattered across the entire data space. Right? Right. Yeah. We don't, we don't want to delete everything that's missing. I don't like the imputation because I don't, I don't believe in it. It's actually, it's actually viable in this space because the, if, if you specify it's it's possible to do imputation the imputation default behavior is just to simply sub each missing one with another record another instance of that variable so that at least you're matching the distribution. I mean, so it's an easy, easy, cheap way to match the distribution of. Huh? Yeah. With replacement of the numeric ones.