Why Blunt Rankings Based on 'ROI' Might Mislead Prospective College Students

Welcome to the future of education. I'm Michael Horne. Looking forward to today's conversation with someone I've known for several years in the education space, Marc Salisbury, who's the founder CEO of Tuition Fit. If you don't know Tuition Fit, check it out. Doing incredible work making the price of college far more transparent to learners everywhere by actually collecting the real data and offer letters themselves from, from actual learners to understand what does a school offer from an actual price as opposed to the list price to a learner to help them make better choices. So they've been doing some incredible work there. But most recently, what caught my attention was Mark put out a piece where he focused on a series of pieces that have come out recently, taking a lot of data from the college scorecard and elsewhere to basically create return on investment metrics for which program at any given college or university is likely to pay off for a learner. And many people, myself included, have been pushing for more outcomes data. And I think it's incredibly important that we start seeing steps this way. But Mark pointed out some really important, shall we say, blaring caveats that we ought to bear in mind. And so with that, I'll bring Mark up to the stage. And first, it's good to see you, buddy. I appreciate you having this conversation. Let's maybe set the scene a little bit better than I did and talk about these return on investment reports that we started seeing coming out where they created rankings in essence of hey, if you do a social work masters at this school, you're likely to be in debt the rest of your life versus this one where you're going to pay X and make this much money and you'll be rolling in money somehow how automatically describe, I guess first those reports that we saw come out and then we can get into some of the problems and guardrails that we ought to have around them.

Yeah, no, it's great to see you, Michael. And it's fun to jump on another version of communicating with the masses through technology. There's always been a set of the population in academia and sort of policy wonks and the like who cringe at the idea of looking at roi. And it seems like that's a sort of fun thing to do. But most people make the case that, you know, it's about, it's way more than money and it's about a lot of this other stuff and you can't measure it without or you can't measure it at all, blah, blah, blah. And I'm not really coming from that side of the camp at all. I'm just saying that, you know, I just started out by saying, okay, when folks are trying to calculate ROI and then trying to convey it, we have a responsibility to be smart about conveying that information and at the same time, not avoid outcomes altogether or pretend they don't exist. At the same time, we don't want to give people wrong information. And so my life prior to Tuition fit was as an institutional researcher at a small college in the Midwest. I've given weeks of my life to reporting this data dutifully to the federal government through IPEDS and the surveys that they require you to complete as an institution. And so what I noticed was the data that was getting used wasn't capable of making the claims that some of these big reports were making. And so I thought, you know what, we better. We better be really careful here. And somebody needs to sort of say something. And, you know, out here on the farm in Iowa, you get a little extra time in your hands, and so you decide, well, I guess that someone better be me.

It's a terrific setup. So tell us. You know, these data, in effect, were basically looking at average returns on salary. Right. And projecting a bunch of assumptions around that.

Right.

But they were also using the denominator, in effect, of what they thought the average price someone would pay for a program and giving you an average return on investment calculation for any given program. What are the problems with that? Because on the surface. Right. That sounds like a noble thing. I would want to that sort of information. What are the problems that you start to see?

Yeah. And, you know, it seems like it should make sense. Right? You've got a whole bunch of data of prices that students have paid, and you have a whole bunch of data of salaries that those students are making now that they've graduated and gone out of the workforce. Well, the. There's actually one way to think about this is that the ROI question sort of matters for two different constituencies. It matters for the institutions because they want to be able to make claims about their quality. But it also matters for the public. And it turns out that the way that matters for the public is different than in the way that it matters for institutions. For institutions, you can just sort of throw all your data together and say, okay, on average, our students are paying X to go to our school, but for the individuals, it's one of those great examples where now college prices are such that nobody's actually paying that average. Most people are paying on either side of that average, and they're paying Enough outside of that average, like 10, 20, $30,000 on one side or the other of that average, that over four or five years that turns into a lot of different money. A lot of, a lot of money, and it can produce a lot of different results. So if you're using that average net price as the I in the ROI calculation, you know, across the institution, that actually, it's not perfect, but it's not horrible. But then if you want to go down another level and look at it by individual program at that institution, you have to make the assumption that one, the average net price is. The range around that average net price is fairly narrow. But then two, that that net price that represents the whole institution is applicable then across all of the different majors. And that's a much harder one to sell because it turns out that across majors, the students aren't equally distributed in the same way that they're distributed at the institution writ large. The other way of saying that is, yeah, it turns out that business majors largely are knuckleheads and physics majors are largely really smart. Right? It's that sort of thing now applied because when the colleges offer the prices that they offer, they're driven largely by merit. And the merit aid is a result of the students academic preparation coming in. And the students that then major in physics showed up with a lot more academic reparation and therefore got a different price on average than the students that were the business majors. And please, to all of your listeners and followers here, I'm really sorry if you majored in business. I'm not trying to knock you off or hurt your feelings, but these are sort of. I'm trying to use the sort of stereotypes that we already sort of have out there, pointing to what I already know from the data that I have seen over the years that I've worked that there's some truth in some of those stereotypes. But that truth then sets up the potential for, as you called it, a blaring caveat when people make these calculations to suggest that a business major from this institution has a great roi and the business major from that institution not so great roi. We're starting with a presumption about what those business majors are paying and we actually have no idea.

So you just said several important things there that I want to dig in sort of one by one on because I think it's worth unpacking this because I confess, when I read the reports, I sort of said, oh, roi, great. And then I read yours and I said, whoa, wait a second. So number one, I Take from it that the average is useful for me to understand perhaps a school at an institution levels quality in terms of this metric. Right. Of in terms of a salary based approach. Right.

Especially if you're the president of that institution.

Right, right.

And you're looking at your own institution. Exactly. Yes, you can.

So I can make that, I can take that data, I can make improvements. I can say, gee, we ought to lower the price, we ought to do more to get, you know, people into higher paying fields, whatever it might be.

Right.

I can make those sorts of decisions at that level. But for me as an individual, maybe less useful at least when it's an ROI calculation because my eye is likely to have a great variability based on merit what that particular college is recruiting for that particular year. We know, you know, sometimes schools are like, we really want to fill the slot with someone from North Dakota. You're not from North Dakota, we're not going to give you that aid. Right.

Like I'm telling you, there's a bunch of schools, if you've got a bassoon player from North Dakota, you are in. Right?

You are in. And you're going to get a nice, you know, nice scholarship. Right. So, and so using those figures, and in an interview we did probably a couple years ago at this point, you made the point that if you just looked at that price, you know, someone, and someone could only apply to one school, say between Boston University, Northeastern and Boston College, they might apply to one of those, but knowing their particular circumstance of their profile, their scores, where they're from, etc. Etc. They, they should be led in a completely different direction because the average is actually unhelpful to them as the individual. So that, that's the first issue to parse out, is that? Right?

That's very important for people to understand.

Okay, so then the second issue I think I hear you saying is that at an institution level the data is probably, or the end size is probably sufficient to draw these conclusions on the average. But when you break down into programs, you two, I think you're saying two things are happening. One, the end size might get a lot, well, actually maybe different from the end size. The spread, the variability might be much wider than is, than you can see where it sort of washes away across an institution. You know, that range might be very wide within a given major. And so you see an average, but you're not actually getting a complete picture of, of the spikes and valleys, if you will, in that average. But, but secondly, I think you're saying also that there's a causation correlation fallacy that people who are attracted to certain programs are or are good fits for certain programs at certain schools, you know, might have different predilections for the, for the jobs that they're going to take, for what, what life decisions they're going to make that are going to qualify them to get certain salaries and things of that nature. And we might attribute it to the choice of major. But there may be more, there may be some deeper characteristics we're overlooking in the individual. So I think those are two issues, but I'd love you to comment on both.

Yeah. That you, you have dug even a little bit deeper in some really, I think important ways there in, in parsing out what, what I'm trying to get at. Essentially take institution College University, like the most generic title school we can come up with, College University. And their sticker price is 50, 60 thousand dollars a year. And the students there are by and large paying between 10,000 and 50,000. So your average is what, 30,000. Right. Okay, but let's take all the business majors. Are they paying between 10 and 50? Well, no, it turns out that the business majors are paying largely between 25 and 55, while the pre med majors are paying between 5 and 30. Why? Because those students, when they applied to college university, submitted their test scores, submitted their unweighted high school gpas and all of their advanced placement courses and everything else. And across those two groups, they looked quite different when they were being evaluated for merit scholarships. And so one group got a whole bunch more financial aid than the other. So those two groups have very different average prices. But you don't have any of that information available when you're doing these big picture analyses that these ROI calculations have done because all they're operating from is the assumption that everybody at that school paid the net price, the average price across all students. But that average price across all students doesn't translate to the average price across each major. And the likelihood that that translates gets less and less the smaller the major. So you were talking about N size, and that's a really big piece of this puzzle because what are the majors that have the smallest end sizes? Well, they're most oftentimes in the humanities and some of the smaller social sciences, maybe in the arts, maybe they're in philosophy, for example. And those are the programs where folks at the board level might say, well, why do we have this program? Why do they have that program? I'm not sure we should have this. And then somebody puts out an ROI calculation That says, well, yeah, the philosophy majors do terribly here, but in truth, it may very well be the philosophy majors. ROI is actually reasonable, if not positive because those students who are also really, really smart kids coming in and they're not paying very much to come to college. University.

Got it.

So we don't know any of that from this.

Yeah, that's super helpful because that's a deeper clarification that I didn't understand before. Right. Which is that we're taking the return on those philosophy majors, but we're not taking the eye on those philosophy majors. We're still looking at the eye across the institution. So that's a huge matching problem that ought to be corrected. Yeah, it seems like the other piece of it. And I'm curious your take on this. It seems like at the graduate level, for example, there's probably huge, you know, ROI calculations to be made on the front end because you're applying to a specific program. Some colleges that's true at the undergrad level. Some it's not true. And so there's a. There's another wrinkle or complication there around the, hey, major matters more than college. I'm curious your sort of take on that. That line that's become accepted wisdom at this point.

Yeah, well, this runs into a pretty stiff problem in that we know that about a third of college students enter college admitting they have no idea what they're going to study. They just don't know they're undecided. A few of them sort of. Well, not more, more than a few of them. But there's quite a few students that even will parse that and say, I'm undecided. But I'll also check off all the programs that I think I might be interested in sort of genuinely verifying how deeply undecided they really are. Of the rest of those students, the two thirds, roughly half of them, begin in a major that they think they are pursuing whether they actually declare that major or not. And then at some point they change majors. So depending on the study, you can find, you know how they count it where it's somewhere between 2/3 and 75% of students end up graduating in a major that's different than what they had in mind when they started. So if that's the case, then an awful lot of this ROI data is not all that helpful for the student. Just period. It's not all that helpful because in their real world experience, they could actually make the wrong choice given what they knew at the time at age 17. And then at 22, they're in a totally different place. They found a different direction. They maybe even found something really deep like purpose and meaning and. But they're at the wrong school for that somehow. So there's just a lot of stuff in here that, you know, requires some pretty big caveats.

No, it's helpful. So one takeaway I'm having is at the institutional level, maybe even at the policymaker level, this data can be useful, but for an individual, we're not at a place where we can take that and act off of it. What would a better solution look like in your mind, to help individuals? And probably, frankly, I think it would also improve policymakers and institutional levels, leaders decision making as well. But let's start with individuals. What would a better solution look like in your mind?

Well, I think. I mean, one thing that would be wonderful is if we actually had student level data, which we currently don't have because the federal government doesn't collect it. And of course, there were all kinds of concerns about privacy. And so we're explicitly forbidden from collecting it. So we don't know the individual prices that students pay. And as a function of that, we're stuck with these really useless numbers. The sticker price that nobody pays and the average price that nobody pays. Right. So student level data would be pretty helpful to begin with. However, I think we still have to be really careful about analyzing the data that we have because it's available and it'd be fun to run a regression equation with your fancy computer as opposed to what is it we really want to know and then work backward from there. So one of the things about these particular studies is we know what students majored in and then we know how much money they're making. Sort of. We can talk about that later on an addendum or something, but we sort of know the money that they're making. So what we do is we then analyze major and then income, but we know that major is actually not really the right data point to assess. What we're really talking about is we want to know student success after college, student life success after college. And some of that absolutely includes being able to pay the rent, save money, and prepare for life as you go through adulthood. But the usefulness of major actually might be a problematic thing there because we know that you can get to a lot of professions through a whole bunch of different majors. And then there are some majors in which it does actually matter that you be in an engineering major to become an engineer or be a teaching major to become A teacher. But that's not the case across all majors. And in fact, you have a lot of students that find their way into a profession after a major that really doesn't even look like it should be related. But then because of the things they did outside of that major, they find their way into a profession. Right. So I think one of the second things that we have to really do is start from, what is it that we really want to know? And I'm going to, without your consent, cheer for something you wrote recently, which was, when we talk about outcomes, we might actually benefit from asking the individual student, what is your intended outcome? You wrote about that recently because a lot of times we don't ask that question. And so we impose an outcome that wasn't even really what the student had in mind. And as a function of that, that sets us off on a policy conversation that might not really help us get to where we want to be in terms of producing the very best possible educational system that produces what it's supposed to produce. Right. So I often think that it would be very helpful for us to sort of step back from all of this, analyze the data that we have and then pretend it says something that I'm not sure it says, and start with, what is it we really want to know? And then what data points we actually need to have in order to answer that question, and then start to really assess this, because then I think we'd be in a much better place.

It's really interesting because hearing you say that, it strikes me that what we really want to do at the individual level is have the do you know what you're trying to do or not? Conversation up front. And as you and I have talked about several times in the past, a lot of students don't. They're going because they're expected to, not because they have any clue. Right.

And because they're 17. Right.

And that's okay.

Yeah.

Right. And that's okay. So put that aside for a moment. If you have clarity around purpose. Great. Let's then look at this set of data points or things that we know around that and help you make a better decision. Because a philosophy major at an Ivy League school, you're probably disproportionately likely to end up at McKinsey and do just fine. A philosophy major at another school without that social capital, perhaps you're not. And that's going to lead to a very different outcome. But we can have those conversations at that level, and then we can ask the question, well, does it. Are you price sensitive, right? Does it matter how much you pay and look at the different packages you are likely to get and make that, make that decision at an individual level as opposed to looking at a ranking, oh gee, I should never be a philosophy major, period, because it just doesn't pay off, period. Which we know is factually untrue.

You said very early in this conversation use the term guardrails. And I think a lot of times what we do with educational data and sort of what we're trying for, it's all coming from a good place. But I think instead of focusing on ensuring that people don't go off the road and end up in real ditches, we spend too much effort on trying to get them right down this narrow chute that they'll fit right into. We got this old Weberian industrial assembly line model thinking. Instead of just say, let's just make sure that they don't end up going off the rails and really get hurt. And if we would spend more of our time there, I think we would leave the room for some of the things that you just talked about. The reality that when you go off to college as an traditional age student, that your middle name should be ambiguity and you should just embrace that because that's the reality of life as a 17, 18 year old, even as a 25 year old male, your frontal lobe has finally come together. That should be part of the conversation as well. And we should allow for that. But with a lot of the data that we have and then the analytics that we're doing, we're suggesting something that I don't even think we really want.

Yeah, it's a really good set of points, Mark. And I would argue that a lot of my business school professors when I was age 25 probably still had doubts that my frontal cortex had fully formed even at that point. So I take the point and I think it's an important one. And like you said, framing this as a risk mitigation, perhaps against the extreme downsides that could really hurt someone for future decisions, that they wouldn't even be aware of their blind spots, but be a little less concerned with over engineering the outcome on the front end before someone is even ready for that conversation.

Right. I think a lot of times we think of life as a straight line and we would love for it to be a straight line and we sort of imagine it that way. And yet we all know it hasn't ever been and it's less so than it ever was now. And so let's prepare people to thrive in the curves, which means just don't fly off the end. Right. Don't fly off over the rail.

Love it. Love that line. Mark, any final thoughts that we ought to bear in mind as we're working through this new data that is still incomplete? I think we'd all love student. Well, obviously there are people who wouldn't love it, but you and I would love student level data records to start to get at some of these. And in my mind there's a lot of protections we could put in place so that they were not used toward ill purposes and violate privacy. But. And any final thoughts around this conversation? Because it's going to only as, you know, continue to swirl and mount in a variety of ways.

Well, thanks. And I think there's, you know, the ROI thing. You know, I think so much of that conversation gets taken over by the. You can't measure ROI at all and nobody should be talking about roi. College is an exploration and just an investigation and go off into the ether and enjoy and think and love and eat, whatever. And you know, that's kind of silly too, in the world that we live in now. But let's be smart about the ROI assessment and let's get it right so that we don't just rearrange deck chairs. We have done this enough in education now that we probably ought to sort of figure out, let's not fumble this one like we've done in some of the other things we've done over the. Over the past couple decades. Let's just be careful and get it right while at the same time encouraging public to think about ROI and be smart about it and play the long game and progress carefully so that to steal from the counseling world, we just do no harm with data.

Yeah, look, I always learn a ton from you. I really appreciate you coming on here to enlighten me, but also a bunch of the viewers and listeners and people who read the transcript here. But I feel like I've learned a lot. It also comes back to have a measure of humility and understand where the average is useful, as Todd Rose would say, and where the average is less useful in terms of individual decision making. But a lot of useful insights here for those watching. Check out tuition fit. If you've got someone going to school or who's recently enrolled, definitely check it out because it can both help with advice, but also frankly, Mark, I'll let you do the last plug, but collecting the information from individuals to help other individuals pay it forward and make better choices is obviously a key part of your model and you pick up a lot of interesting and sometimes scary things in that world that I think are important for folks to know.

The best way for us, I believe, to move higher education to a better place because we need it to be good. We just do. We can't, we don't blow the whole thing up. We need higher education to work for as many people as absolutely possible. The first thing that we just need is transparency, clarity, be able to see into the mirror and see what we got. And the way to do that, really. The colleges, they're probably not the first movers in this one, but us as a public, we can create the solution ourselves. And once we have this data, this data can help us answer so many of the other questions and break up so many of the log jams that are holding back innovation at many of the institutions that we need to innovate the most. So I hope folks will spread the word about Tuition fit and join us in this project to see if we can pull back the curtain and make price transparency a reality.

Terrific. That's a great place to leave it. And for all those watching, thanks again for tuning into the future of education. And if you like this conversation with folks like Mark, give us a thumbs up. We'll have him back. Because, Mark, you're always pushing my thinking. Deeply appreciate it and we'll see you next time.

Thanks, Michael. It's great to see you.