Calculating the ROI of Higher Ed

This is Future youe with Jeff Salingo and Michael Horn. And welcome to Future youe. I'm Jeff Salingo, and I'm here today with Marty Vanderwerff of the Georgetown University center on Education and the Workforce. It's great to have you on Future youe, Marty.

Thanks, Jeff.

And so tell us a little bit about the Georgetown center on Education and the Workforce, because we see a ton of reports. We see the head of the center, Tony Carnevale, quoted often around workforce and education issues. Tell us a little bit about what's its mission and what are you trying to do?

Well, the mission is to really talk about the value of college, what is needed in the workforce, and to really measure it quantitatively the best we can. So we're always looking for data. We're always looking for ways to describe how college is valued in the workplace and the skills that you learn in college and how they're valued in the workplace. So we might do reports, for example, about if I get a degree in electrical engineering, what am I likely to make? That sort of thing. We also look at equity in higher education. That's a big mission of ours because we feel as though college should be serving more of the population, a greater cross representation of the population, and we feel like it doesn't do a very good job of it.

So you recently put out a report on the ROI of thousands of institutions, the return on investment, and kind of ranked them in some sort of order, which was a little different than many of your reports, which are more kind of quality. They use numbers, but they're more at the higher, at a higher level. You don't really name names as, as often in, in your reports. Can you talk to us a little bit more about that report and how it came about?

Well, we discovered, not too surprisingly, that rankings get attention. Jeff.

Imagine that.

And, and if you go, if you're just talking about higher education in the abstract and you're not talking about specific institutions, there's no accountability. And we feel like there should be more accountability and more transparency in higher education. We feel like the college scorecard data, which is where we drew the rankings from, is a real attempt to give consumers the public transparency and accountability and more information about colleges. And so we felt if we didn't rank them, then you wouldn't actually know what the value is of an individual institution. And so what good would it be?

Well, you know, this whole concept of ROI of a college degree is pretty controversial in higher education. The belief is you should go to college to Get a broad education and you'll, you'll make it throughout life if that, if that happens. Why should we measure roi?

Well, I think, Jeff, you among many others have been very good at chronicling the cost increase in higher education and the fact that people just don't have the time or the money to go and explore and find themselves in college anymore. It really has become a place where you go to get a credential that's going to help you get into the workplace. And so I think colleges need to embrace that. I think that the data is out there. The data is only going to get richer. They need to embrace this opportunity to sort of talk to their consumers about what it is that they're getting from the education. It's not a question that's going to go away. So we just decided to face the question straight on.

Now talk a little bit about the methodology, how you did this. You mentioned you took data from the college scorecard, but I know some of the college scorecard data is limited. You were able to look beyond some of that data by taking some calculations without getting too in the weeds. Could you tell us a little bit about the. How the methodology behind this and how you did it?

Yeah, really very simple if you think the best way to think about it, Jeff, is that the lifetime roi, the net present value, is essentially your lifetime earnings minus the cost of going to college. That's the easiest way to look at it.

And so you rank these colleges, and when I was looking at the list, you had a number of colleges that had great return on investment. You had a number of colleges that were at the very bottom, of course, but you had kind of this huge, huge middle of institutions. And what fascinated me in looking at that middle is that you had all kinds of institutions we wouldn't expect to kind of come together yet. Privates and publics. Four years, two years for profits. Community colleges. When we think about the American higher education system, we tend to think of it in tiers, but we tend to think of privates as different from publics. Four years is different from two years, but all of them kind of were stuck together in the middle 50%. What are we supposed to take from that?

Well, you know, the value proposition in higher education is mostly built on price and selectivity. And so it's a natural instinct for people to think that the more selective college they go to, the more they're going to get out of it. And so when we looked at that 50th percentile, the surprising thing is we found a lot of colleges that people would tend to think of as being very different. Community colleges for profit institutions, regional publics, selective privates, all giving you approximately the same return on investment. So, you know, one, one college I, I use as an example is, is Oberlin. Oberlin is a, is a very selective private, hard to get into, but also has a music conservatory, which is sort of at the core of what it does, among other things. Now, people who go to that music conservatory, like most music students, hopefully have very, very productive lives, very happy lives, but they may not make a lot of money. And really, obviously our main metric here is earnings. So, you know, the one thing that we can't tell you is quality of life from the metric. What we can tell you is earnings. And what we can tell you by looking at all those 50th percentile schools is how many different schools, how many different schools at how many different price points give you approximately the same output.

So is your hope that would be students and in some cases, if we're talking about 18 to 22 year olds or parents as well, use this as one tool in a toolkit in terms of deciding college. In other words, you shouldn't decide college just based on this, this ranking.

Absolutely. You know, I think about this because what are people looking for when they're looking at a college? Well, they want to go to the one that has their major. They want to go to one that they can get into. They might want to go into one that's nearby. They want to go to one that they know a little bit about. And those things are always primary. Obviously they want to go to one they can afford. But on the second tier, probably of things that they might consider as data becomes more and more available, is what am I going to get out of this college? Is this college going to give me a payoff at the end? So we think eventually this data will come into the conversation. I don't think it's there. Now we're going to go in with our next report too. We're going to start looking at programs within colleges. So a lot of the data that we have now, and I could go back to Oberlin as an example. Now, not everyone who goes to Oberlin majors in music and not everyone who, who goes to Oberlin is going to make the median roi. Like all colleges, different graduates of different programs are going to make different earnings. And so when we get into this program level data, you'll actually be able to compare majors within a college, but also an English major at this college and an English major at that college and at that college, et cetera. And you'll have a much broader array of data. And so we're going to go into that next.

You followed this report with one that specifically looked at liberal arts colleges, which are really kind of quote, unquote, under attack now or they feel under attack because they feel like their programs are not seen as producing the job outcomes that parents and students want. And we've seen a huge race away from some of these majors. And in fact, at some colleges and universities, right. The humanities are in huge decline and the science is coming up. Computer science is, you know, the new English major. So what did you find in the follow up report where you look specifically at the ROI of liberal arts colleges?

Well, so we, when we looked at liberal arts colleges, we found that they have, frankly, a pretty surprising roi. They rank only behind large private research universities or large research universities that are not all private. We thought there's a lot of reasons for that. Frankly, of the data of the set that we looked at, almost a quarter of them are pretty selective. So they are.

So the inputs are pretty good there.

Exactly. And we also know that when you look at liberal arts colleges that I, a disproportionate number of students go on to graduate school. So since we aren't able to separate out who went on to graduate school, but we, we know empirically, we know from the data that a lot of people who go to, say, Carleton or Kenyon go on to law school, medical school, graduate school of some sort. So they probably have a higher earnings level than most other students.

Well, Marty, thank you so much for joining us on Future youe Today.

Thanks for having me.

And we'll be right this episode of Future you was made possible with support from the Academy for Innovative Higher Education Leadership. The Academy is a partnership between Arizona State University and Georgetown University and is the premier training ground for those who aspire to senior leadership positions in higher education and those who want to lead organizational change at colleges and universities in the future. And welcome back after that conversation with Marty Vanderwerff of Georgetown University center on Education and the Workforce. I'm now joined by my co host Michael Horn through zoom. Welcome, Michael.

Thanks so much, Jeff. Good to be with you as always in this medium, even in these times.

Yeah. And so we should note, Michael, that we actually recorded that interview with Marty before the COVID 19 pandemic shuttered campuses worldwide and before George Floyd's death, death in Minneapolis. But I think it still has incredible importance and resonance in today's Movement when you agree.

Yeah, I totally agree, Jeff. And listening to it, I was struck by thinking about to the last recession where we know that millions of Americans went back to higher education. Enrollment soared at a lot of institutions and there were also a lot of disappointing outcomes. Right. As debt without degree increased. And so I kind of think that the important conversation you had with Marty is so important because we need to give students and regulators right now better tools to help students make choices consistent with why they are going back to school with a real return on investment. Focus. And I think, you know, you mentioned George Floyd, of course, this is our first podcast recording since that incident occurred. Black students in particular, out of the last recession were disproportionately impacted by choosing schools that didn't have good track records or didn't deliver on the promise. And obviously for profit universities have shouldered a lot of blame for that. But it wasn't just the for profits, frankly. And if we fast forward to today, black unemployment has skyrocketed in ways that are nothing short of tragic, literally on the heels of, you know, black Americans experiencing record low unemployment right before the pandemic. So we've had a stunning reversal, Jeff and I. You know, we know that many are likely to return to higher ed, to upskill and reskill. And I just think we need these kinds of tools right now to help understand return on investment and allow them to make sound choices, which is the very nature of the conversation that you had with Marty before all this happened. So his, his message and his research feels even more important now in some ways.

Yeah. And I think that's a really important point. I was happened to be interviewing recently Karen Stout from Achieving the Dream, which of course is a nonprofit organization that works with community colleges across the country, primarily around institutional redesign, largely around student success initiatives. And one of the points that she made to me was that, you know, we clearly have moved the needle on student success efforts, at particular at the two year level, which were incredibly low in some cases. Right. So you mentioned, you know, for profits, which have been widely publicized for their, for their poor outcomes. But you know, we should throw some community colleges in that bucket as well for that reason. And one of the points she made, which I thought was interesting, is that for so long we've talked about student success overall, but we haven't talked about the equity gaps within student success. And you know, all boats have been rising within the student success efforts in community colleges. But the point she made was that there are still large equity gaps between African American students at community Colleges, for example, and white students at community colleges. And one of the things she hopes comes in the coming years is that we start to look at those equity gaps more closely to figure out what we can do on the student success efforts within community college, within community colleges. I also think we have some data to back up how important this conversation is right now. We know as we record this episode or the follow up to this interview with Marty, that the STRADA Education Network has been conducting weekly polling on Americans experiences and attitudes in relation to education and work amid the pandemic. And what STRADA has reported, for example, is that African Americans and Latinos are more likely than white Americans, for example, to have changed or canceled their education plans in the midst of the pandemic. These individuals are most likely to have delayed enrolling or reduce their coursework. So that idea, for example, that Karen was talking about of these inequity and these gaps that we see in retention and completion perhaps could even get worse during this pandemic. In addition, Stratus found that African Americans and Latinos are more likely than white Americans to enroll in education and training programs in the coming months across learning providers. But compared to white Americans, a larger share of African Americans and Latinos plan to enroll in online programs, work based learning programs, community colleges and four year colleges and universities. And so as a result, again, I think one of the things that we have to figure out as we look at the ROI is to really dig a little deeper into these, into these equity gaps. So Michael, I'm curious, you know, you've spent a lot of time thinking about outcomes from individual programs and institutions. It's been really a big topic of your writing for a decade now. You're also chairing the Educational Quality Outcome Standard Board, which many of our listeners may not know what that is. So maybe you could talk a little bit about that because that's focused on creating these standards for audited reports on student outcomes at the programmatic level.

Yeah.

So what are the big things that jumped out to you in listening to the conversation with Marty?

Well, I think, you know, the point that you just raised Jeff around, disaggregating by race and by income level and by gender and a variety of characteristics is going to be really important to dig beneath sort of the average, if you will return on investment statistics. No Child Left behind in the K12 world is sort of used in a negative way generally when spoken around. But. But I think it did a lot of important things in the K12 world, which was create transparency into the actual Outcomes because people just didn't have the data before and it disaggregated by subgroup, which was incredibly important. I think we need to see the same thing in higher ed, to your point there. And you know, look, the Education Quality Outcome Standards Board, which we created when, when we were at Entangled and which is now part of Guild Education, so it's a separate nonprofit now that our friend Kristen Sharp is leading. You know, it's so important because it's starting to create that trail and we are disaggregating by subgroups. So I think it's incredibly important to give that level of transparency, not just for accredited programs, but all of the unaccredited programs we know people are going to flock to in these times as well, like boot camps and so forth. I think a second thing, I guess there's two other things that sort of hit out at me. So a second thing, Jeff, would be that I think income share agreements, something that we have profiled a few times on Future youe, I think is another important part of this conversation because it's the notion of institutions sharing in the risk with students who have taken out debt or you know, with those typically taxpayers who put upfront capital to finance students going back to higher education. I think having institutional risk is going to be really important as we think about the design of programs to get Americans back into higher education, allow them to upskill and reskill. And then I guess the last thing that I'm struck by, and this is the counter notion to everything I just said, but it's. I do think we ought to have some humility at the same time as all this. And just because there are some limitations to the data, data, data approach, namely that data is by definition, right. It's inherently backward looking and it, it only becomes really convincing once you have a lot of hindsight to it. Right. And you can scrub the data and understand what it's saying. And, and by that time, the programs you're looking at may have evolved to be very different from what they were, say, you know, a couple of years earlier. And then the last piece of this is even within the subgroups, no one's going to just adhere to the average, you know, like you might say 13% of black Americans graduate from community colleges on average. Well, you might be in a different situation where it's really the right program for you at the right time, or vice versa. Right. And so I think being able to somehow know that the data, it is going to be limited, even as I still think it's an important tool so that we can start to make some more informed choices at the very least. But I'm curious, you know, your take, you obviously had this conversation directly with Marty at the time. You were excited about it at the time. But I'm just curious, you know, your takeaways.

Well, I think there's two things in the data that he talked about, one positive and one that I worry about. One is we're clearly now getting to the ROI on higher education at a much more substantial level that I think helps consumers. Right. Michael, We've both written a lot in the last couple of years and both of our books are really focused on consumer decisions around higher education. We've talked a lot on this show and everybody has talked a lot about how the rising cost of college and how students and parents are making decisions. Adult students as well, are making, you know, one of the largest purchasing decisions in their, in their life with a lack of data. So it's clear now we're starting to get, you know, we used to think higher education's worth it because a bachelor's degree recipient, for example, makes so much more money than a high school graduate over the course of their lifetime. Those averages were terrific when higher education was free or close to free for most students. That just doesn't cut it anymore. And Marty made reference to that in, in his, in his interview. And so now we're able to get these program outcomes both at the institutional level and obviously now at the program level in terms of students major. I'm really excited about the next movement down, which is at the skills level.

Right.

We both know, in fact, just talking to our friend Ryan Craig again in the last couple of weeks, who talks a lot about kind of skills based hiring and the friction in the hiring market with higher education increasingly employers and I think coming out of this pandemic are going to be looking for skilled employees in certain areas and they're going to want to know that those people have those skills. We know that we used to think of a degree as a proxy for that. We used to think of an institution as a proxy for that. Well, if they went to Yale and they majored in X, they must know something. And I think increasingly we're skeptical of that.

People listening to this podcast are very skeptical of that in my case. Right, right.

So that's why I picked Yale. So, you know, so I think if we could get down, if we could get down to the skills based level, if we could talk about the return on investment of individual skills within programs, within institutions that to me is kind of the holy grail here in terms of what we're after and at the level of data that I think would be useful to have, particularly for students making these decisions. And that, by the way, might be even more important for adult students who may be taking time out of the job market in order to go back to school. So that's what I see as the positive of this push to roi. Where I see the negative is exactly what happened to the US News and World Report rankings starting back in the 80s. And you've had a chance, I know, to pre read my book coming out in the fall, who Gets in and why? And there's a lengthy discussion in there using Northeastern University as an example of how institutional decision making has been impacted by the rankings over the years. Northeastern being one of many examples that I could have used where institutional leaders, and this is a previous regime at Northeastern than the current president where they basically reverse engineered the rankings to figure out how to rise in them. And institutional decision making on everything from faculty to students and who to enroll and who to recruit was all based on gaming. And I don't use that necessarily in a negative way on gaming the rankings. And so my concern is, is that as this, you know, better data comes out at the program level, at the institutional level, and even at the skills level is how are institutions going to react to that and are they going to do the same thing? And so, for example, would they get rid of, you know, because maybe the returns on investment aren't great in the humanities, and so will they get rid of humanity? Will that increase the flight away from the humanities and thus institutions cutting those programs? Will they invest in programs where today's skills, the ROI and today's skills are critical, but we might forget about kind of the next iteration of programs that will come out. I just worry that because we lack so much knowledge in higher ed around outcomes. And again, this is a positive. This is the positive part of this. The other side of the coin that I worry about is that institutional leaders don't make decisions in a vacuum. And I worry about them trying to do what they've done with the rankings over the years, and that is to try to make themselves look better.

Yeah, I think it's a really important point. I just want to actually hit on two things, Jeff, before, before we sort of close out this episode that you said. One, on the negative side of institutions making decisions based on the data, I'm sure some of our listeners are saying, well, if you have the right data like, isn't that a good thing? But I think the point is, what a lot of political scientists always point out is that there's always unintended consequences once you sort of pick a number or even a set of numbers. And just to make it super clear here on your humanities example, it may be the case, for example, and I think we know that this is often the case, which is that your first, second, third year salary when you graduate with a humanities major is lower than those with STEM majors. But if you look 10, 15 years out, you catch up and sometimes surpass those with STEM majors. Right. And so it's, it's very easy to, you know, pick a short term measure, right? Measure salary after you graduate maybe a couple years after 10 years out, that's going to be really difficult. Right. A lot of variability and you know, to my worry about the data lagging, that's going to be a huge lag if we're talking about a decade of measurement. Right. That, that, that we're sort of making decisions on. The second thing, I think that's interesting because you brought up our friend Ryan Craig. It's sort of another way around this quandary, if you will, which is Ryan is big, not just on last mile training programs that are hyper focused on skills, but then programs that actually hire those students to stay with them for a year or two afterwards as temp workers so that employers can, quote, try before they buy to really de risk that. And I think programs that do that, you know, independent of the data conversation we're having programs that are willing to put their capital and their, you know, effectively their money where their mouth is, really backing their graduates in that sort of a way that'll send a signal over time. And that's also the programs that, that, that students ought to be looking at increasingly. I'd be curious your quick reaction on this. I'm not sure higher ed, traditional higher ed is going to mimic what Ryan's hoping for there.

No, and unfortunately I think that we need to a point where students are not only graduating with a degree, but that they're also graduating with some workplace credential.

Right.

Ryan makes the point, I think all the time of what is it? Salesforce or whatever SaaS platform is the requirement of many jobs. And he asked the question, the rhetorical question, how many institutions teach that? And the answer is zero. But what if in addition to your English degree, you got a certificate in that SAS platform as part of your four year degree? Would that then help your ROI over the long run? Because not only did you get that English degree which supposedly provided you those soft skills employers want, but then they're providing you this very specific hard skill that employers want and are required in a job. And by the way, the employers don't have to train for that and it could potentially increase your first year salary and your second year salary and make the learning outcomes from that institution institution look much better.

Yeah, it's so these are all good things to keep an eye on. We're going to start to see partnerships. You know, I know Pathstream that provides that, for example, is partnering with institutions increasingly. So all things to keep an eye on, to be sure, at a critical time in our nation's history and in the future of higher education. But that does it for this episode. And so thanks to Marty for joining us again. And most importantly, thank you all for listening. We love hearing from you. So please drop Jeff and me a line with ideas, comments, questions, or of course, even complaints. Until next time, stay safe and stay strong. Hey folks, Michael Horn here. Hope you enjoyed the latest episode of Future youe. And just a reminder to please subscribe to us on your favorite podcast platform and if you like the podcast, rate us so that others can find us and find out about the good conversations that we're having here. As always, thanks so much for listening.