Metrics that Matter: Counting What is Really Important to College Students

Welcome back to another episode of This Week in College Viability. My name is Gary Stocker. Today we're going to speak with Dr. Zachary Bleemer, who is a co-author of a recent book that just came out, and it's entitled Metrics That Matter. We all know that high schools and college students are inundated, inundated by indicators and rankings supposedly designed to help them choose a college. In the new book, Metrics That Matter: Counting What's Really Important to College Students, we have with us today one of the co-authors, Dr. Zachary Bleemer. Dr. Bleemer, welcome.

Thank you very much. Good to be with you, Gary.

And we have a list of questions that we want to ask, both to help us understand the book better and to get a handle on how both students and their families and maybe even colleges can use the work that you and your colleagues have done to make the college selection process, I guess, marginally better, to make it easier as appropriate for students. And to start off with this, if you wouldn't mind, Dr. Blimor, just share What's the history behind the books, behind the book Metrics That Matter?

Absolutely. So, you know, this book has 5 co-authors. It has an unusually large number of authors. We are all academics from across the spectrum of the university. So there's an English professor, 2 qualitative social scientists, and 2 quantitative social scientists like myself. My background is as an economist. We all gathered a couple of years ago. We spent sort of 3 months pretty sequestered together on the campus of the University of California at Irvine. We gathered really to just try to talk across disciplinary boundaries and try to understand first how kids are making important choices about whether to go to college, where to go to college, and what to study when they're in college. Second, which metrics were most important to those students and how are those metrics constructed? So things like undergraduate rankings or sticker prices of tuition, average wages by college major, literally what data were going into these well-known metrics. Then how are students using them, and were they actually useful in the ways that students were manipulating these metrics? And what alternatives might exist that would be more helpful to kids as they were making these choices? And what we found is that bringing all these different disciplinary perspectives together was really helpful to try to understand these different points of what went into metrics, how are they being used, and what kids could use alternatively. And that, that's, that's sort of resulted in the book. The book has 7 chapters and each chapter is about a single metric, just sort of leading people through where this metric comes from and what alternatives people should probably be looking for instead of this very well-publicized information.

And I want to kind of take a look at most of the chapters in the book and just ask questions from each chapter. And we've heard the term return on investment or ROI, and I think college leaders and academics and those those in the higher education industry use it all the time. What about from a parent's perspective? What are you talking about in the book that helps a parent and their student better understand ROI?

Yeah, so there are a lot of things that college does for a person. It provides labor market benefits, but it also provides critical thinking skills, ethics and morality skills, and sort of ways of thinking about the world. It provides long-run health benefits and, you know, a wide variety of other ways in which college really reshapes the lives of its students. ROI focuses in on one of those returns, which is the impact of going to college on the financial health of a student. And the idea of a return on investment indicator is that a kid should be able to plug in information about themselves, you know, where they come from and some information about their family. Some information about what college would cost if they were to enroll at college, and then what wages they would receive if they went to college. And the ROI is supposed to spit out like a thumbs up or a thumbs down. Is it worth it to invest in college for you, some particular 18-year-old, at this price and at this level of return? This seems like a really promising way of just sort of answering the question of whether college is financially beneficial for young people. What we found, though, is that when you actually try to look into what available ROI metrics exist and how are they usable by students, there's just a lot to be desired, so much to be desired, that it's really not clear whether available ROI tools are at all helpful even to answer this very narrow question that they are defined to answer, which is whether college is worth it for a young person. So the first chapter of our book is really the primary chapter focused on whether kids should go to college in the first place. And we dig into a couple of these ROI measures. I think the best known ROI measure is constructed by a company called PayScale, where you can just go onto their website and they provide average wage returns and cost information, and then, you know, do this, run this through an algorithm to tell you what's the value of going to some college major in some university. The limitations here are really numerous. One is that the data that PayScale is building its ROI measure out of are self-reported data from a relatively small sample of people, anyone who happens to have gone to this website before and directly reported their wage, alma mater, and major to PayScale. There are a lot of cases where very few people have reported that data for a wide variety of universities and programs. And even if a large number of people have reported, it's a totally non-representative sample. It's just the people who parents who have gone in and provided information to the sperm. There are a number of other ROI programs, some of which use data from the College Scorecard, a federal database, and some from large-scale private surveys. But in all cases, the data are largely unrepresentative or long enough term to represent the kinds of kids who actually want to use these tools to make important choices. And so our takeaway after sort of digging into these ROI measures is that even for someone who just wants to figure out whether college is worth it, well, these are not a helpful way of going about that. Now, we have a very large set of academic research that the chapter goes through that suggests that the returns to 4-year universities for nearly all students are very strongly positive, except in cases where costs are really exorbitant, where kids have to pay many tens of thousands of dollars to enroll at a private university. But especially in the cases of public universities, there's rarely a question of whether the human capital and other skills provided by four-year schools exceed the costs faced by students who enroll at them. The value of universities are really tremendous. And so after this first chapter, we then focus in on these questions of where one ought to go to school and what one ought to study while one's in school. Taking a step back from this question of whether college is worth it in the first place, our sense is that available metrics aren't helpful, but there are a few kids for whom college is important.

Interesting. And then we've heard many of those arguments before about ROI, but in particular on pay scale, it's such a limited sample size. You're making a judgment based on not much data at all.

Yeah. And unrepresentative data. Even if the sample size was small or was large, it still wouldn't be representative of the kinds of kids who are trying to use this information.

We hear the term— to go on to the next question— good college, best college all the time. And of course, that is most commonly associated, Zach, with university rankings. And if you follow higher education at all, like I'm sure you and I do, and many of the folks who listen to our podcast and view our channel, you know, U.S. News and World Report is under a little bit of pressure, but they're always in the news because their representation of good and bad while making large business sense doesn't always provide consumers with what they need. How do you address that in your book?

Yeah, so I think there's this question with university rankings of what objective they are serving, or what the goal of a ranking is in the first place. It's not clear, for example, whether rankings are being produced for students or for institutions or for faculty or for parents, there are good reasons to expect that those different objectives would lead to different rankings. There are schools that are better for students, schools that are better for, you know, so you'll hear, I'm a faculty member at Yale, you know, it's an amazing place to be a faculty member, but just because it's an amazing place to be a faculty member doesn't necessarily mean it's a great place to be an administrator or a student, et cetera. When you think about what a university is providing to students, it's providing labor market opportunity. It's also providing tremendous skill acquisition that can be used to promote innovation, that can be used to promote entrepreneurship, that can be used to generate mobility for lower-income students. It's not at all clear that a university that's good at one of those things would also be good at all of the other things. And so when you look at the sausage that goes into rankings, You look at the US News algorithm, which they make publicly available, you can see the weights that they place on all these different pieces of information. But it's, it's hard to even think through what available justifications could be provided for the specific weights that are provided, that are assigned to each of these components. Instead, I think when parents and students are trying to decide which university is going to provide them the greatest values, they should just be looking directly at what universities are doing for students and the degree to which universities lead their students to success. So what we do in our chapter on university rankings is hone in on a couple of really high-quality academic studies that try to understand what would have happened if a student had gone to a different university. Instead of going to, you know, actually the case that we focus on in the book is instead of going on went to a relatively selective or a high-performing private university in the state of Massachusetts, what would have happened if they went to a flagship public university in that state instead? What we find is that kids who are making choices between relatively similarly ranked universities, if you were just to look at the US News and World Report ranking, those kids end up having very similar long-run life trajectories. It doesn't really matter if you go to a slightly higher or slightly lower ranked school. But it does matter in terms of likelihood of actually earning a college degree and in terms of longer-run outcomes like going to graduate school or how much money you earn in the labor market in your 20s, 30s, or later. It does matter if you go to like a much higher-ranked university than a much lower-ranked university. So to give one example, you know, focusing on this case of Massachusetts, there's this really nice study of the Massachusetts Adams Scholarship, a scholarship program that continues to run by the state government and public schools, public high schools in Massachusetts. What the study looks at are kids who become eligible for this Adam Scholarship. You have to be in the top 25% of your high school as ranked by like the Massachusetts state standardized test to be eligible for the scholarship. And if you get the scholarship, you can go for free to a public university in the state of Massachusetts. And so what you do is you look at kids right at the bottom of this eligibility pool, people in the 25th percentile of their high school class. They like barely made it in. To the Adams Scholarship Program. And the study compares those students to students just outside the Adams Scholarship Program. If they had, say, a B+ instead of a B in their 10th grade English class, they would have made it into the Adams Scholarship, but they didn't, for which reason they had much less of an incentive to go to public Massachusetts universities. It turns out that the schools that kids went to when they received the Adams Scholarship were pretty substantially lower ranked. Than the private universities they would have gone to if they hadn't had access to an Adam Scholarship and where they would have received financial aid from the institution directly instead. So this is sort of an interesting financial aid program. It's actually incentivizing kids to go to relatively lower ranked or somewhat sort of heuristically worse universities. And it turns out that you can better characterize the degree to which those universities are so-called worse by their graduation rates. So kids who didn't get the Adams Scholarship went to private universities with relatively higher graduation rates. Kids who got the Adams Scholarship went to public universities with lower graduation rates. And when you follow up with those kids 5 or 6 years later, you see that the kids who went to the lower graduation rate school themselves were less likely to earn a college degree. Going to a higher graduation rate institution makes kids themselves more likely to actually make it through a school. I think This isn't always obvious to people. You might think of high graduation rate schools as being more selective, more difficult. Kids are going to have a hard time competing at those schools. But actually, going to a school with a higher graduation rate seems to provide pretty substantial benefits, not just to kids in terms of their likelihood of actually graduating and earning a degree, but if you then follow them for years later, it looks like going to a higher graduation rate school provides relatively larger labor market gains for these students. So what we recommend in the chapter is rather than looking at this sort of sausage of university rankings that can be theoretically used for all these different purposes, if instead you just focus on graduation rate differences between schools— there are lots of schools with relatively similar graduation rates, and that's indicative of something. Those schools are providing relatively similar value to their students. But there are also cases where schools have radically different graduation rates, and when those big differences exist, it seems like going to the higher graduation rate institution seems to provide relatively substantial value to students.

Interesting. And I'm going to do a follow-up question on that in a second. But one of the things that, that I've done in the College Viability App is I, for private colleges throughout the country, I track both the 4- and 6-year graduation rates.

Right.

And I'm sure you've looked at that, and in general the numbers are distressing.

Yeah.

But, and so I kind of want to change your career for a second, Dr. Blumer, instead of a college professor I want to make you a college counselor just for a minute or two, right? And keeping in mind this university rankings, as a college counselor, what one tip would you give parents and their students about using college rankings?

Like I just said, it seems to me that rankings provide no information to students about the relative quality of the university over and above what's already provided by a school's 6-year graduation rate. What I mean when I say that is, let me give you a different example because it's a setting that I'm somewhat more familiar with. I spent a lot of time in California studying the California university system. So California has this very hierarchical system. There are 9 undergraduate University of California campuses at the top, and then 22 California State Universities that sort of function as the local comprehensive universities for the state. A large set, over 100 community colleges. And when you look at the relative graduation rates of these schools, what you see is that even though people have like a very strong sense of like a sort of ranking of the University of California campuses, like Berkeley and UCLA are at the top, and UC Riverside and UC Merced are relatively lower, the graduation rate differences between the University of California campuses aren't that substantial when you look at 6-year graduation rates. Berkeley and UCLA are higher, than the other campuses, but especially when you look and sort of excluding those two schools, the differences are relatively small. Whereas when you compare those schools to the graduation rates at the California State University system, in some cases you see really large disparities. There are CSU campuses with graduation rates that look a lot like the UC system, and then graduation rates that are far, far lower, 30, 40 percentage points lower in the six-year level than the University of California campuses. Those differences are the differences that really matter. So as parents are making choices between institutions, it's those big differences in graduation rate that people should be paying attention to because those really are indicative of the long run, the degree to which universities can provide long run gains to their students.

And then I want to stick with this topic because you're making some points that I think have value. I want you to put your college professor and researcher cap back on because I've seen the arguments that those make when their college has either public or private, has low graduation rates. And a common argument— you heard this, I'm sure— is, well, we accept everybody, so our low graduation rates don't matter. How do you respond to that?

It's a really interesting question. So, you know, take two universities. One of them has a 75% graduation rate, the other has a 50% graduation rate. And you wonder, okay, there are like at least two plausible explanations for those differences in graduation rates. One is that the 75% school graduation rate school might do a lot of work to get its students to graduate. So it might provide student services, it might provide substantial value that sort of motivates students internally to persist in the field. There are all sorts of things the school might do, and so it might be sort of the treatment effect of the school that leads 75% of its students to graduate. But there's an alternative story. The alternative story is that the 75% school rejects two-thirds of applicants, whereas the 50% school essentially admits everyone. And so if you were just to take the students from the 75% school and put them into the other school, well, maybe 75% of them would still graduate. Maybe it doesn't matter where those particular kids go to college, but they're just, they're better students. They're more likely to persist no matter where they go. And so it's not a treatment effect of the 75% university, but it's instead just a selection effect. They're just choosing kids who are going to graduate no matter where they go. I think that's a really persuasive kind of understanding of graduation rates. And therefore motivates research questions of like, what happens when you take a kid, the same kid who would have gone to a school with a 50% graduation rate, and instead gets pulled into a school with a 75% graduation rate, right? If the story is treatment effect, then that kid's going to become 25 percentage points more likely to earn a college degree. Whereas if the story is selection, it doesn't matter where you go, then that kid's going to go from a 50% chance to a 50% chance. It's not going to matter that they go to the higher graduation rate institution. There are now a series of, I think, quite high-quality academic studies that are effectively doing just that. So one study that I was very involved in studies a program in the state of California called Eligibility in the Local Context. This is a top-percent policy. It guarantees the top 4% of students from every high school in California access to most University of California campuses, basically all of them except for Berkeley or UCLA. It was running during the 2000s. It largely ended in 2011. But for 11 years, the top 4% of kids get pulled into schools like UC Davis, UC Irvine, UC Santa Barbara, et cetera. And so what you can do, sort of similar to the Adams Scholarship policy I told you a moment ago, you can study ELC by finding in a school of, say, 400 kids, finding the ranked 16th and the ranked 17th high school senior from that high school. The kid who had the lowest GPA that still put them in the top 4%, and thus who got access to a school like UC Irvine, and then kid number 17 who was not eligible for the ELC policy. They weren't in the top 4%. And so most of those students ended up going to a local comprehensive university, one of the CSU campuses instead. And so you can just take all these pairs of students. Every high school that ran this policy, they did it for 11 years. They have 1,000 high schools in California. So that's 1,000 times 11 times these two pairs of students. And then we just follow them 10 years and see what happens. Who earns a college degree? What happens when they join the labor market? How much money do they make? And what you see is that going to UC Irvine instead of the CSU campus, which is exactly what I just described, it's a 75% school instead of a 50% school, increases the kid's own likelihood of earning a college degree within 5 years by 20 or 25%. 10 points. It's almost 1 for 1 with graduation. You see the same thing with earnings. The kids who get pulled into schools like UC Irvine are making on the order of $20,000 a year more by their late 20s relative to the kids who are still going to the CSU campuses. They are basically their local comprehensive university instead. So there's now a number of studies like this. And so even if you would expect that graduation rate plays this dual role of treatment effect and selection, It turns out that graduation rates seem to do a really good job of indexing which schools provide value to their students.

And it ties in really with the, the next question that talks about selectivity, the chapter in your book on selectivity. And I want those listening, we're talking with Dr. Zachary Bleemer, who is one of the co-authors of Metrics That Matter: Counting What's Really Important to College Students. And Dr. Bleemer, some really good stuff already, but let's go to the chapter on selectivity. And for those of us that follow higher education, Jeff Selingo is a popular and relatively well-known writer, and he wrote the book Who Gets In to Colleges and Why. And he made the case that it doesn't really matter what college you choose, it's what you do with that college experience that matters. How does your book position college selectivity?

Yeah, it's a great question, and I think it really highlights this difference between schools that are relatively similar to each other and schools that are very dissimilar to each other. So, you know, as I was just talking about graduation rates, you know, if you look at, say, the top 50 ranked universities in, like, the U.S. News and World Report, these are all schools with below 20% or sometimes up to 25%, uh, admissions rates. So they're very highly selective institutions. They all have very high average SAT scores. They all have graduation rates that are quite similar to each other, in the range of 80 to 85%. With some public universities that go as low as maybe 70 or 75%. These are schools that sort of by the numbers are very similar to each other. And I think that sort of the primary motivation, the empirical study that Jeff Salengo is sort of pointing to when he says, oh yeah, it doesn't really matter where you go to college, is a study that really focuses on those top 50 schools and points out that the difference between rank 20 and rank 40 is essentially nothing in terms of long-run outcomes. There's no evidence that the relatively more— the higher ranked university provides any additional average benefit to its students. But that's not true when you compare schools that are very different from each other. So I was just giving the example of the 75% versus 50% graduation rates. Once you're looking outside the sort of cohort of top universities, or the set of University of California campuses, or the set of flagship public universities, there are increasingly substantial differences in quality between institutions. So what our chapter on selectivity goes through is again sort of reflecting on these differences by graduation rate and also points to a second metric that pretty sharply distinguishes the kinds of education that different schools provide by focusing on instructional expenditure per student. So unlike graduation rates, which are really easy to look up for lots of universities, expenditures on instructional activities per student at the university level are a little bit more difficult to find. You have to go to the College Scorecard, this federal government website, they have pages for each university, but those pages for some reason don't include instructional expenditure per student. You have to literally go and download the underlying data. It's a huge Excel spreadsheet that records lots of information for every college and university in the US. And one of the columns in that spreadsheet provides instructional expenditure per student. This is literally the amount of money that schools are spending to educate their students. And again, what you see is a difference between small differences and big differences. If you were to look at the top 50 private universities in the US, you'd see some variation in expenditure per student, but really not very many, not very much of a difference. Whereas if you look at a flagship public university and a local comprehensive public university, you'll see very substantial differences in graduation rates and also in instructional expenditure. And we think these are relatively high-quality ways for parents to get a sense just a big picture of which schools are doing a better or worse job or providing more or less to their students. Once you start looking at small differences, I think we totally agree with Chad. There's no reason to sort of focus on these small differences, but big differences matter. And there's no reason to look at rankings for those big differences. Big differences in graduation rates or instructional expenditure sort of gives you all the information you need to know about the degree to which universities provide long-run benefits. Students.

Interesting. And you talked about these— the spending per FTE. And one of the things that I track on the College Viability app is that very item. Is it spending per FTE for institutions, academics, instruction support, and student services? And interestingly, the thing I look for is, for the most part, they don't vary. My study goes for 8 years. They don't vary that much. But it's those times when the spending drastically changes that you say, uh-oh, something's going on there. They've made some kind of business decision.

And you can see that either in instructional expenditure or student service expenditure, there could be pretty drastic changes in both. And both, I think, I agree, are worth paying attention to.

And I've seen some in the institutional support for FTE too. We see that in the literature, spending more on non-academic resources. Interestingly, not many folks like to track that because it seems kind of too much inside baseball kind of numbers. But nonetheless, yeah, nonetheless, it's a fascinating topic. The next chapter in the book talks about tuition sticker price, and I kind of want to set this one up a little bit, Dr. Bleemer. And we're all aware that tuition includes— the tuition prices that colleges, both private and public, offer includes what colleges call scholarships or merit aid and what you and I know for the most part are unfunded discounts. And so in your book, when you talk about tuition sticker price, and there's a reference that I've seen, I can't remember the name of the organization, but tuitionfit.org. Mark Salisbury is one of my good friends, and many other organizations are promoting the comparison services of what colleges offer. If this type of service becomes more widespread, how do you see it impacting colleges in general, selective colleges, private colleges?

Yeah, it's a fantastic question, and I agree. I think TuitionFit is a really awesome service that is increasingly providing high-quality cost information to students. You know, what we focus on in the chapter is first pointing out, as you've already said, and I think is well known in the higher education community but is less well known outside that community, that while tuition costs have been rising 8-9% a year on average for the last 30 years, that totally confuses what's actually going on with the cost of college, which on average has hardly changed in that entire period. For a middle or even upper-middle-income student because very few people actually pay the sticker prices that are advertised by universities. The sticker price really should be thought of as like the maximum price. The university is prohibited from charging any more than that price, but the large majority of students who are going for most private and indeed many, many public universities are paying far less than those sticker prices. And so what we do in the chapter is first discuss the degree to which the crazy sticker prices that are being publicized by many private universities have discouraged relatively lower-income people from ever sending an application in for college in the first place. We have pretty— the research literature has provided a lot of evidence that lower-income parents have much worse information about what college would cost for their kids than high-income parents. In fact, if you just run a nationally representative survey, look at people in the bottom quarter and the top quarter of incomes in the United States, and you just ask them, what would college cost if your kid were to go to college at a public university in your state? The parents of low-income students report a higher number than the parents of high-income students, despite the fact that parents in the bottom quarter of incomes in the US their kids would, in almost all states, just go to college for free. There would be no tuition price at all. Whereas kids in the top quarter of incomes would pay either the sticker price or something relatively close to the sticker price. So in other words, despite the fact that low-income parents would pay much less to go to college, they believe college to be more expensive for their kids than high-income parents believe it to be. This is totally backwards. And so what we do in the paper— sorry, what we do in the book is make a couple of recommendations to students about ways that they can try to learn what college would actually cost for them. And TuitionFit is one fantastic example. We highlight another, MyIntuition, which is sort of a second-generation price calculator that's been made available by many of the sort of— it tends to be on the more selective university side where TuitionFit tends to be more valuable for the middle selectivities. Sector of universities. But whatever information or whatever tool kids choose to use, I think it's really important to try to make that information as easily accessible to high school students as possible. And the key thing is just really just totally ignore these sticker prices that totally get in the way of people evaluating what college would actually cost. It's, you know, you should think of college as a good that's purchased in the marketplace. And this is a marketplace where prices are obfuscated, where The salespeople are doing their best to make unclear to potential consumers what it would cost to purchase a good. Any economist will tell you that this is a setting in which it's really hard for consumers to make informed choices. And providing this kind of information to kids is, I think, really crucial in growing the number of students who choose to make this, I think, really important valuable investment.

And if you think about it, both higher education and healthcare are both industries where the consumers really are so uninformed by comparison to other industries. And yet they're the two most consequential decisions that all of us make during different parts of our lifetime. It's, it's, and one of the questions I have, and I'll probably get to a little bit later, I'm going to jump ahead a little bit. And again, at College Viability App, we are, we provide a service, an app that we use the IPEDS data. To be able to let anybody compare the financial health, the outcomes, the enrollment of private colleges mostly, although we do some work on public colleges as well. Is it reasonable from your opinion or from what you and your co-authors did in your book, is it reasonable to use a college's recent financial and outcome enrollment history as a selection tool?

That's a great question. So what we focus on in the book is sort of a cross-section, like what's the most recent data for a university and what can we learn from that data? What you're pointing to is these trajectories, you know, how are graduation rates changing or how are instructional expenditures per student evolving over time? When you look at most universities, you see that those things don't change that much year over year. I think you mentioned this just a moment ago. And so at schools where those trajectories are relatively flat, there's little reason to pay much attention to them, just focusing on the most recent data I think they're sufficient. But you're also right that there are some schools that are changing pretty dramatically. And I think students should be somewhat wary of those schools, no matter which direction they're trending. It just means that these schools are out of equilibrium. And those quick changes make it difficult to know exactly what the experience would be like to be there for 4 years. But so I definitely think that's something worth paying attention to. You know, our, our sense is that as, you know, parents who don't want to spend too much time on this process, or students who are interested in going to college and are just trying to get a first sense of, you know, what's the landscape, where should they be paying attention to, what would it cost, uh, where should they go, our sense is that, you know, things like graduation rates or the immediate institutional expenditure at these schools, you know, the, the use of, uh, as high quality a tuition calculator as is available to get a sense of what it would cost that particular student to go to school. Our sense is that those are the sort of first-order things that people should be paying the most attention to. But in cases where these trajectories are, uh, are pretty sharp, yeah, I totally agree it's something that it would be really useful, at least to a second degree, and maybe at some schools in the first degree, uh, for parents to take some action.

And yet, as I look and, and market that College Viability app, there are precious few— and you're not going to be surprised at this— that precious few parents who even think that a college's viability is in question. And yet when we look at the horror stories at the Concords, at Mount Ida, at McMurray's here in Illinois, it's, it's, it's almost crushing to read those. And the market will adjust, I understand.

And, you know, they made these choices and in some cases lost years of their life.

Yeah, yeah, yeah. So one of the IPEDS data fields is admissions yield, and I won't bother to define that for you because you know what it is. And I've added my own little descriptor to that, calling it a popularity indicator.

Indicator.

Yeah, because really, is— do you think that the admissions indicate— the admission yield, excuse me, is a reasonable popularity indicator? Because students have said, hey, I'm going to choose that college that accepted me, or not?

It's a great question. I think there's two things going on. One is that universities are extremely conscious of their own yield rates, right? They're basically the admissions office at most selective private and many public universities this is the primary metric that they're targeting to try to understand how many students they need to admit in order to yield the size of the incoming class that they want. And so as a result, admissions offices are paying a lot of attention to, for example, whether they should admit some given student, not just because, I don't know, they think that student is qualified for the university, but well, maybe we should reject them because there's no way they're coming to this school in the first place. And so there's a sense in which because admissions offices are in many cases rejecting students not because they're not qualified, but because they just don't believe those students have a sufficiently high likelihood of yield to make it even worth it for them to admit that student. In those cases, you're going to have this sort of biased measure of popularity. It partially measures popularity, but it also partially measures the degree to which admissions offices are doing their best to curate their admission pools. To just identify students who really want to go to the university and admit those students. So it's sort of, you know, I think the first thing to say is it's sort of a popularity measure, but it's manipulable in the same way that many university statistics are manipulable. So, you know, you might wonder, like, okay, why would a university want to manipulate its yield rate? And a key reason is that the university has an interest in admitting as few students as possible because that makes the university look more selective. A key metric that lots of people use to determine the relative quality of university for reasons that we think are confusing and unhelpful. But nevertheless, these things are important. And so while universities are trying to manipulate selectivity, as a result, they're manipulating yield rates in a way that might make it confusing to interpret. So I think that's the first point to make. The second point to make is, OK, let's say yield was a measure of popularity. Would it therefore be a good way of choosing where to go to college? And it's not at all clear to us that popularity is a good way of figuring out which are the high-quality colleges or which colleges will provide the greatest value to students. So, you know, again, this is a case where we recommend things like graduation rate and instructional expenditure, which seem to do a really good job of indexing the long-run gains that universities provide, whereas yield, even as a measure of popularity, is probably just not the most helpful available metric to figure out which universities are providing value.

Moneyball, and I presume you're aware of the book, started the era of data entrepreneurs in baseball. And as we see this year with all the changes in the rules in baseball, I'm trying to project that mindset for higher education because I know from my perspective, and granted, Zach, I live this on a daily basis, is it possible that the growing set of higher education data entrepreneurs like yourself, your co-authors, myself, and many others, will impact this industry just like Moneyball did baseball?

I think it's a really interesting question, Gary. So here's at least the way that I would think the question through. I would first start by highlighting an important difference between baseball and university. So baseball has a well-defined notion of productivity. Teams have a goal. The goal is to win games. And so what Moneyball did was it brought in this influx of data to a baseball team that was able to use that data to really closely fit or to really carefully make changes to maximize productivity. And it was really successful in this. Data are really helpful once you have an objective in mind. In achieving that objective. The difference in the educational sector is that there's no well-defined notion of educational productivity in the sense of like, what would it mean for a university to do an amazing job for its students? Well, one thing that universities are doing is they're educating kids. They're providing lots of education. But data can't— well, data can help you provide more education, or you can imagine data helping professors, teachers, uh, uh, do a better job of educating students. But data— there's no well-defined notion of productivity in terms of admission. So like, which are the best students for a university to admit? Well, the university might have different objectives. If their objective is to admit the most capable students, the students who are going to do best in school, and you can imagine them using SAT scores and high school GPAs, what high school people went to, AP classes, etc. If that was the goal of a university, then additional data could help the university achieve that goal. But it's really not clear why that's a reasonable goal for a university, especially a public school that has a public mandate. Why would public schools go out of their way to only educate kids who already had the best education before age 18, just to provide those kids more education? There's just no well-defined productivity notion here. Alternatively, you could imagine universities that are going out of their way to find the kids who will learn most at those universities and target those kids, admit those kids. Data will help you identify those students, but it's really a non-trivial exercise. You can't just use available metrics of applicants to figure out who would learn most. That's a much trickier task and a task that I think academia is very much still grappling with, trying to understand who gains most from university access, selective university access, access to certain college majors inside of universities, etc. So it's not clear to me that additional data in this sector would change, say, undergraduate admissions in the same way that analytics changed baseball, because there's just no obvious goal that data are going to be used for. Instead, there's many goals. Universities have lots of objectives, and data will help them achieve many of those objectives, but it seems to me it's going to look very different. Than the Moneyball book, where, you know, we went very quickly from a no data to a very data-heavy environment. But everyone's kind of using data in the same way. There are certain well-defined statistics. You want to maximize those statistics. There's little reason to think that higher education would evolve that way because everyone's objectives are different.

So the final chapter, I think it's the final chapter in your book, Dr. Bloomer, is wages by major. And your book discusses how debt decisions, student debt decisions, are deeply personal. And that it creates a focus on cost for higher wage earners. And many of the recent news stories I know I have read have been about colleges cutting majors and/or some combination of courses. And most of these— this is a subjective analysis on my part— most of these can be classified as comparatively lower wage majors. How do you and your co-authors see this focus on high wage majors impacting those colleges, those degrees, those programs with comparatively lower wage projections?

Yeah, yeah, it's a great question. So wage by major statistics only became widely available quite recently over the last 10 or 15 years, but are now extremely available. You go on what we call a scorecard, and for almost every major at almost every university, you can see the average wages of graduates who have earned that major. And it's clear that students have responded to that information. We have pretty high-quality academic research that shows that when you inform students about the relative wages in different majors, you see students shift their preferences toward higher-wage majors. The highest-wage majors, things like engineering and business and economics, but even toward relatively higher-wage majors, and studying political science instead of studying English, because political science tends to have graduates that earn more money. What we talk through in the book is how students should understand these wage by major statistics. First, where they come from, literally what data are being used to estimate them, and while there are a wide variety of sources, none of those data sets are quite what you would want them to be. In most cases, there are sort of odd groups of students who are omitted, maybe students who drop out and so never actually earn a college degree, but those students are still relevant to a freshman who's deciding what major to study because different majors might have different dropout rates. You know, these wage-to-major statistics exclude homemakers or anyone who doesn't have a formal sector job in the U.S. despite the fact that employment levels are going to be really important to people as they're making major choice decisions. So there are a lot of nuances, and the book goes through many of these nuances and how to understand wage-to-major statistics. But The book then focuses in on, I think, a really fascinating study that tries to sort of take everything into account and understand what happens when a student who really loves, say, the humanities, they really want to be a history major, but they're getting pushed to study, say, geology instead because they hear that geology has higher relative wages. What happens if the student differs from their passion and goes and studies something they're less excited about? but has average higher wages. And this is a study that came out 6 or 7 years ago in an economics journal. It focuses on a kind of interesting setting. It's outside the US. The study is focused on the country of Norway. So Norway does undergraduate admissions quite different from the US. In Norway, when you're a high school senior, you take an exam. You have your high school grades. And then you send into a centralized national admission system a rank order list of here's the most preferred major and school pair that I'd like to study, here's number 2, here's number 3, all the way down. And then what's going to happen is that there's going to be a centralized admissions algorithm that takes those preferences and grades and scores into account and then assigns each student in Norway to one program, the program that admits them, and they're admitted directly into a university major pair and hopefully they get into their most preferred or more preferred major there. So what this study does is compares a couple of groups of students. It looks at kids whose first choice is history or some other humanities major. Some of those kids get into their first choice major, but some of them don't, and so they have to go to their second choice major. And so we can focus on kids whose second choice major is a STEM field. So what happens to kids who want to study humanities but have to study STEM instead because that's what they're given? And Then you also have a different group of kids, kids who want to study STEM but get pushed into humanities instead. So they rank a STEM field first and humanities field second. And you can follow these two groups of students for 10 years later and see what happens to their wages when they get pushed between these majors. And what the authors find is, I think, really interesting. There's one piece that's sort of unsurprising, which is that when there are students who really want to study STEM, but they get pushed into humanities major instead. If you compare them to the kids who were able to study STEM, the STEM majors have higher earnings. So kids who prefer STEM, who want to earn STEM degrees, and who are able to earn STEM degrees end up with relatively higher earnings than the kids who are pushed into humanities. All right, that's not surprising. How about the kids who want to major in the humanities, but some of them get pushed into studying a STEM field instead? And there, it turns out, you see the opposite story. You see kids who get pushed into STEM fields don't have higher wages. If anything, they, they too have lower wages than the kids who are able to study their first choice humanities degree. And we think this is really interesting. Again, there are cases of big differences between wages— English and engineering. In those cases, getting pushed from English to engineering raises kids' wages. Engineers make a lot of money even if they don't want to be engineers. But when it comes to the humanities and the sciences, that's not true. Kids who get pushed into the sciences from the humanities in Norway don't make any more money than they would have made if they were just allowed to study the humanities as they wish. And that's just because the average wage gap between the two is relatively small, and preferences matter in this context. It seems like giving— allowing kids to study what they most prefer leads to labor market benefits that don't accrue if they have to study something else. That they really weren't as interested. That's our takeaway from this, this study, or rather our chapter on the wage by major statistics. Small differences not just don't matter, but might actually mislead people. They might lead people to study things that they care about less, and that might actually lead them to lower average wages than the lower paying majors they were otherwise going to receive. Large differences matter, though small differences Speaking with Dr.

Zachary Bleemer from Yale University and one of the co-authors of the new book, Metrics That Matter: Counting What's Really Important to College Students. And for those who have an interest in either personal or professional interest in taking a different look at the way we measure what colleges to choose, what colleges' impact has, and what may be the future of colleges in general, Metrics That Matter: Counting What's Really Important to College Students is a book you want to put in your library, be it hard copy or on your ebook, to read in the coming future. Dr. Bleemer, our pleasure. Thanks for joining us today.

Hey, thank you so much, Gary.