Data Mislead with Bob Moesta

Hi, everyone here with Bob Mesta, the author with me of Choosing College. And I wanted Bob to come on because there's just been questions that have come up as I've been out on the road about the book and things of that nature that I just thought, why not ask the expert? So here you are.

Well, ask us both. What are some of the questions?

So the question I want to tackle this time is, everyone says, gosh, if higher ed just had more data, it's so hard to figure out outcomes of colleges, like how students actually do. If, you know, when you searched Google, for example, there was just data galore. If the Department of Education collected more data, people would make better choices. We'd have better outcomes. I have a feeling, I mean, we've both said this, that that's not really true. Why not?

What's wrong with that? I don't think so. There's a couple things. One is that data is very abstract, and the aspect there is like, there's lots of different kinds of data. And what Clay and I always used to talk about or talk about was the fact that most of the time, we measure what's easy, not what's meaningful. And so we end up with a lot of data that actually doesn't tell us anything. And we need to actually spend more time understanding what are meaningful measures and how do meaningful measures work. And so, to me, part of it is being able to get to, what does progress really mean? And so, for example, somebody who's trying to help me, help me get away. Right. The reality is, how do you know they're making progress? Well, progress for them is like, look, I just want to get out. I don't know what else to do. I'm okay with that. But over time, we should see progress being like, I know who I am. I know what I'm good at. And then it should be something along the lines of, not only do I know what I'm good, I have some options that I'm looking. And look, I went and found my path. My thing is, we have to figure out how to create new metrics. Most people think of metrics as data and data points, but the reality is, real, meaningful data is hard to measure.

So, in other words, we need to situate the data that we collect or the metrics that we collect within the context of what someone's actually trying to.

That's correct. That's correct. And so what happens is, Todd Rose has a great talk about this, where he talks about the difference between when we average something, we literally realize that the column of data is the average of the column, but that one data point is actually in the context of the row and clusters make actually way more sense than just averages. And so the second part of this is, even with more data, we need different ways to analyze that data to make sense of it. Right? And then there's a third part of this, which is that there's like the timing of the data. There's what we call a lead measure and a lag measure. So like, so the fact is, like, if I want to lose weight, right, the lag measure is my weight. The lead measure is the calories that I actually eat or the type of calories or the type of the number of hours. So part of this is to realize that like, not all data is the same. And so this is actually a really. It's not necessarily a more complicated problem. But the reality is you need a really good frame of that context to basically start to realize what is the most meaningful data.

So this helps to explain that even when we have data, we tell a student, well, if you go to college A, on average, you know, X percent graduate, if you go to college B, you'll have this much data and only this much graduate. And then they choose college A. And you're like, why did you just do this? This actually helps to start explain that because maybe we're not framing it.

I think it's totally explained. I always think of things happen at an individual level. And the fact is those causal mechanisms are very, very different. And so what happens is because we can't explain it, we put it to error. And so to me, the other part of this is the fact is, the amazing part is, I think I'll say the overuse of statistics where if I just increase the sample size, I can actually figure out whether something statistically significant or not. But. But if it's the wrong question, more sampling more of the same thing doesn't help you actually answer the question.

Yeah, so it's interesting because we have a lot of policy right now from, you know, state departments of education, department of education and so forth, they want to get more outcomes out there. They want to measure graduation rates and things of that nature. It's interesting in the language of our book, if you're like, and help me step it up and you can exit the program early and get a job, that's why you actually want it.

That's actually. That is success for them. Like, you get penalized. Yeah, well, I mean, school gets right. I did an interview with somebody who basically wanted to learn, learn a new programming language. So they gave a better job and they literally took the class and in the middle of the class they got the job offer to basically get the job. And they got the rest of the training at work. And in their mind it was like, that was exactly what I want to do. I wanted to go learn the language I got. But the reality is like the school's like he dropped out.

And all the traditional metrics, all the.

Transitional metrics would say like, you know, you failed with that student. And the student's like, no, like it helped me get the other job because I had start, I, I couldn't have started the path any other way.

So it's just a reminder that it's not just about the data, it's about actually the data in the right context.

That and then there's one other point which is there's difference between descriptive data and causal data. Right? And descriptive data you can talk about like who I am by describing me like I'm six foot one, I'm £240, I'm et cetera. And that describes kind of who I am, but it doesn't describe what I do. And so the data that is connected through space and time is really what's most important in terms of understanding how people make decisions. Because what happens is what I did yesterday or what I did last hour or what I did last minute has impact on what I did today. And so part of this is understanding where people are in space and time. And when I say causal data versus actually just descriptive data, that's super helpful.

And it also says the people that just want more data. Well intentioned. But intentions aren't everything. We got to get the causality right.

What is it? What is it? What, what's the rule is like, you know, the, the, the, the road to hell is paved with good intentions.

Yes, yes, exactly. So we, we will try to avoid that trap. We'll cut it off here. But thanks for joining us.

Thanks.

Thanks Bob.