In the Wake of the Admissions Scandal, How Much Does the SAT Matter?

This is Future youe with Jeff Salingo and Michael Horn.

Welcome to Future youe. I'm Michael Horne, joined by my co host, Jeff Salingo, and really excited for today's conversation, we have Josh Goodman, an associate professor at the Harvard Kennedy School, studying human capital and education policy. And, Josh, welcome to the show. A question, Josh, that we love to ask our guests when they come in is how do they get. How did you get interested and started in higher education more generally? But in your case, coming from the discipline that you did, it seems like more and more professors bring that background right now. And so how did you get into that?

Yeah. So not to make it too long a story, but I was a physics major as an undergraduate. Couldn't figure out what to do.

I didn't see that coming. Okay.

Couldn't figure out what to do with my life afterwards. Got lucky, won a scholarship to go to Cambridge University for a year to study anything I wanted. And the only other thing that interested me was education policy. I'd always been very interested in actually in teaching. I'd loved being a teaching assistant in college, and I'd always been interested in sort of debates about education policy. When I did this degree where I studied with sociologists, which was, for me, a big shift because it was words instead of numbers. Came back to the US Was a high school math teacher for a couple of years, and then realized that I wanted to do something that took all the math skills that I learned as a physics major and that I felt like were sort of part of the value that I brought to the world, but apply it to questions that I cared about. And the questions really were about education policy broadly. So I went and did an economics degree because that felt like the intersection of sort of mathematical thinking, but applied to questions that I cared about. And higher ed turned out to be a great place to do work because there's just increasing amounts of rich data to do this work on. And, you know, that's where our skills come into play.

So one of those pieces of data. You published a study recently in August, looking at data on millions of SAT takers and analyze the outcomes of students that did and did not retake the test. I'm kind of fascinated by the SAT and the ACT because I'm in the middle of writing a new book on college admissions, and I'm inside the admissions process at three universities. Emory, Davidson, so small liberal arts college, big private, selective, and then University of Washington, big public. And I'm really interested in how they're using the act and SAT standardized tests in admissions. But tell us what you found and then I want to dig a little deeper on that research.

Yeah, so this was prompted by there've been a series of research papers looking at what happens when states mandate the SAT or the ACT as part of their accountability systems. And what it turns out that does is it brings a whole bunch of students into the test taking population who never would have taken the test before, and some of them end up going to college who wouldn't have otherwise. And so there was all this evidence that getting a kid who wasn't going to take to take it once did something for them. And we wanted to say, well, there's this other thing that people do which is they take it once and then they take it again, and then sometimes they even take it again and again and again and again and again. And so the College Board was very generous, was willing to share their internal data with a few of us researchers. And the first thing we noticed was in their data, there's a huge income disparity in who retakes the sat. So high income kids, much more likely than low income kids to retake. And that immediately tells us that there's, you know, if high income kids are doing something, almost certainly there's a reason that they're doing it because it's giving them some advantage. And so what we set out to do was study the question of does retaking the SAT actually help you go to college, get into college, go to college, go to a better college? And without getting into the details too much, the answer is yes, it does seem like retaking makes a big difference to kids, admissions, relevant SAT scores. And that in turn seems to get them into colleges that, you know, better colleges than they would have otherwise gotten into. And let me just say that I think the fact that was so striking to us, the question we started with is only about half of students who take the SAT retake it. And that struck us as really surprising because it turns out when you look at what colleges say they do with SAT scores for admissions, they super score them. 80% of them say they will use the maximum score that a student gets. It could be, you know, could be the maximum score you get in one sitting or the majority of them across sitting too. Like we'll take your best math and your best reading and put them together. And so it basically cannot hurt you and can only help you to retake the sat. And yet half of students aren't doing it.

And the other thing, Josh, is that I think that sometimes people Forget that, you know, these are high school students taking it. Their brains are just consistently growing over time. And one of the things I've been interested in around kind of early decision, the increase in early decision at colleges, especially very selective colleges and universities, is that for many students, they kind of have to be done with their testing in their junior year of high school. Right. So did you also see that when you start retaking it, you're going to take it over a period of growth in your own kind of educational ability? Right. So that's another reason to take it again because in six months you could grow a lot in terms of your abilities.

So I think that's absolutely true. So I think first of all, you can just learn by taking the test once what it is like to take the test. So separate from even getting smarter, you're experienced. Exactly. There's almost certainly some growth going on because you're studying in school. That may give you more time to study out of school for the exam. And let me just say that the timing of the test turns out to be this really interesting thing where again, there are huge disparities by income and race and ethnicity in when people first take the sat. So higher income kids first take the SAT usually in March or May or June of their junior year. So if it doesn't go well come the fall, senior year, September, October, they take it again and it's fine, it's in time for admissions. There are a lot of low income students and underrepresented minority students who first take the SAT in senior year, which means that they don't even have time to retake necessarily before college admissions deadlines happen. And so we found that actually the timing of that first take is the variable that most explains whether a kid will try to retake the exam again, more so than income.

And do most states that require this, require it early enough so that there is time to retake?

That is a great question, and I don't know the answer to that. Those requirements kicked in a little, a little bit too recently for us to do a lot with them in our data.

But yeah, but it's fascinating also. So that means high schools. I mean, my recollection of high school, and it was a while ago, but was that the school was sort of, you know, as culturally, if you will, or they were preparing us to take it in that junior year. That sounds like that's not the case. I think a lot of these schools

that low income students attend, I think that's absolutely right. I think, yeah. Whether it's culture or just lack of information. I will say one of the things we did for this paper is we did a little bit of letting Google autocomplete fun studies. This is what I call sociology on the cheap. Economists don't often talk to people, but we go on computers to see what people in total are talking about. And so if you type in things like retake the sat, the kind of Google autocompletes you get are, can I retake the sat? So there are lots of students who don't know. You also get things like is retaking the S.A.T. bad?

Bad? Sure.

And it turns out when I've presented this research in front of audiences, I've had some people, particularly the older people for whom they took the SAT farther back in time, say explicitly that they were told that retaking would look bad for their college applications because it's maybe a sign of weakness. Whereas now it seems to me that the advice should be completely the opposite, that as long as colleges actually do what they say they're doing, retaking is a good idea.

Josh, where does this research fit in with the big test optional movement that's happening in colleges and universities? Of course, the University of Chicago, probably the most selective place this past summer that went test optional. How does that fit in with this kind of movement?

Yeah, I think the really broad question this all raises is what do we want colleges selecting students on the basis of. And in particular, if we're worried about access and equity issues, do we think that tests like the SAT in the end are barriers to that?

Or.

But as an economist, I always want to say you got to think about what the alternative is. So if Chicago's not judging people by SATs, what are they judging by? What are they judging? Is it the list of extracurriculars or restaurants, which in some ways may be easier for higher income students to get an advantage on the sat? And I'm not someone who's out here shilling for the College Board, but for all its flaws, it's a fairly straightforward thing. We can argue about whether the SAT should make it. The College Board should make it easier for students to take the exam in terms of lowering the fees even more for low income students, making it easier for them to do that. But. But I think there's an argument to be made that it's not clear whether going test optional actually is something that improves equity and access.

Well, and they did a lot also by with Khan Academy to give free test prep away to low income students, so that somewhat leveling the playing field from the high priced tutors that high income students get. Did that make a difference on this question?

You know, I'd love to know the answer to that. I don't yet know the answer. You know, the one thing that comes up when I give this talk, you can look at how other countries do standardized testing. And the US is somewhat unusual in that we give multiple chances. You know, the SAT is offered seven times a year. We could move to a, you know, a system like China's or Brazil's where you get one shot and that's it. And if it doesn't go well, you know, either you're off on a different track for the rest of your life or you have to take a year off right. For study, which, you know, has. It's not clear that that's, you know, that world is a world you want to move to.

No, I mean, it drives a different type of after school studying culture till hours of the night, quite literally in places like China and Korea and so forth. So switching gears a little bit because there's another piece of research you've done that has gotten both a lot of attention and fascinating around the Georgia Institute of Technology's MOOC powered, if you will, degree. And just for folks that aren't familiar with that research, can you describe what you found there and what you were trying to look at when you came into it?

Yeah, about five or six years ago, Georgia Tech, which has a top 10 computer science department and a very prestigious in person master's degree program that costs about $45,000 takes a year to do, announced that they were going to create a fully online pathway to that same degree that students would do part time, but for only about $7,000 in total. And it would lead to literally the same degree. Not a degree that had the word online in it. They said, we will teach you exactly the same stuff and certify that you with this online learning know as much as our in person students do. And so people were surprised when this happened in part because all the initial excitement about MOOCs radically disrupting higher education had died down a bit. People were sort of looking for sort of positive news, let's say, and there were concerns, you know, why hadn't other universities done this? Well, why wouldn't this cannibalize the in person revenue stream that Georgia Tech had set up with its traditional program? And so what we found in this paper, I think was two interesting facts. One is that the set of students enrolling in the online pathway to this degree was basically entirely different from the in person set of students. So the in person students that Georgia Tech was getting were basically straight out of college, very heavily foreign born, although not entirely a lot from India and China. What the online program was attracting was mid career Americans. The average age of the online folks applicants was something like 34, 35, a decade older than the in person folks. And it makes complete sense that you know, who does online education appeal to?

People in general. I was going to say it's typically your 35 year old.

Exactly. People who have a home, a job, family, they're not going to be mobile or be able to take a year or two off to study. And so one thing I think Georgia Tech figured out was how to in some sense separate its customer markets, which was, I think very clever. So that was, that was interesting. Just from a descriptive. Who is this appealing to? And then what we really set out to study was for these mid career Americans, if they didn't get into Georgia Tech when they applied, what were they doing otherwise? And the answer was basically nothing. It was either this low cost where they are. Yeah, exactly. It was either this low cost, you know, pretty prestigious option, or basically nothing else in the existing marketplace seemed to appeal. And they were doing little bits and pieces of online training, but nothing that in total added up to the amount of training that was going on through this program. So, you know, that I think the real amazing punchline that we came up with is that because Georgia Tech was basically creating new master's degrees, that something like that. Georgia Tech is now responsible for increasing the number of American master's degrees in computer science by something like 10%. That one program is producing 10% of American master's degrees.

So it speaks to something different actually, which I've been fascinated with, which was this supply, demand, imbalance in computer science trained, let's not say even master's students just trained people in the country relative to the jobs that are out there for them. We've seen boot camps come into this space and fill a bunch of that as well. But still the demand and supply are out of whack with each other. And so the cannibalization question almost seems secondary. Why aren't other universities doing this? Or do you think they'll take your research and actually now get into it?

So this is a great question. The economist to me says there's at least one thing that would make me hesitate if I were another university, and that's the following. According to Georgia Tech, it cost them something like 2 to $300,000 per course to create a high quality online course that they felt would do what they needed to do pedagogically to make this degree meaningful. So in total, I think they said, you know, they needed a startup grant of something like $4 million that I think was given by AT&T as a corporate partner. And so that kind of cost structure, that huge upfront investment, only works if you then can scale the program to a place where, you know, because the marginal cost of adding another student is close to zero. So you have to distribute those fixed costs over all those students. So if I'm the second mover in that space, I think things are a lot harder because Georgia Tech is already there. And at some point, I don't know how much space there is. I think there are real economies of scale here. And we know that in markets with economies of scale, there's natural monopolies. Yeah, I don't know.

So, Josh, you talked about in the beginning, the data in higher education. What are some of the other questions that you maybe give us a little preview of some research you're working on or not, but what are some of the other questions that you think need to be answered in higher ed or that you're interested in pursuing, given the data that's out there?

Yeah, I'll say related to this online education research. Our goal with this Georgia Tech paper at first was just to say, what is this doing in terms of the education people are getting? Is it substituting for other things or adding to it? But I think there are interesting longer term questions about what are the returns to these online degrees? It's a little early to answer those questions because Georgia Tech only started graduating people a couple of years ago. But we're working with them to actually find out what the earnings trajectories of their applicants and enrolled students were. So I'm hoping in a year or two, we might be able to shed some light on whether this degree is actually valued by the labor market. I think there's also this fascinating rise of micro credentials that no one has yet. I mean, again, these are very new, so it's early days, but that no one has really got a clear sense of. If I go online and do one course through, I get a little certificate that I learned from one course or a four course. Micromasters. What do employers make of that? Do they take those things seriously? It feels to me like that's a frontier that would be nice to head down. And I'm hopeful that we've found some research partners who have an interest in answering Those questions as well.

I want to ask you about dissemination of your research. When I started covering higher education 20 plus years ago, you know, you'd have to wait for the journal articles to come out, you'd have to call up the researcher, they call you back three days later, things like that. Right. You're part of kind of a new era of researchers that is very active. For example, on social media. I love when you're coming out with research, you do these great tweet storms kind of explaining your research in layman's terms. Is that now a part of how you think about dissemination of research? You and others, I guess, other colleagues?

Yeah. I think early on when I was a faculty member, I sort of had the opinion, well, if you do good research, people will just find it.

Right.

And that's not how the world works. You have to do some self promotion. And I also think you have to be careful as a faculty member. Our immediate professional incentives are to get this stuff published in journals and get the respect of our academic peers. Because when it comes time for tenure, that depends entirely on what other professors know about your peer reviewed published research. But, but the writing for other academics is not often super helpful for practitioners and people who want to disseminate this stuff into the world of policy and practice. And so I found that frankly, that Twitter is actually a great place that sort of forces me to come up with the shorter, you know, so what version of the paper that doesn't necessarily involve some of the details that actually do make the work better, but can be buried in an appendix.

And what's your Twitter handle again?

Oshuasgoodman.

Okay, great. So everyone should follow Josh. And Josh, thank you very much for joining us on Future youe and talking a little bit about your research. We're looking forward to what's coming next and we'll be back after this break.

This episode of Future youe was made possible with support from the Academy for Innovative Higher Education Leadership. The Academy is a partnership between Arizona State University and Georgetown University and is the premier training ground for those who aspire to senior leadership positions in higher education and those who want to lead organizational change at colleges and universities in the future. This episode was also made possible with support from Entangled Solutions. If you want to shape the future of education, Entangled Solutions would like to hear from you. Entangled Solutions is hiring smart, mission driven team members interested in helping world class institutions solve their most vexing challenges in learning and education. Learn more at Entangled Solutions.

Welcome back to Future You. Michael Horne here And Jeff, coming off a great conversation with Josh Goodman, a lot of things to pick apart there. But I'm curious, what jumped out to you most about his research on the sat, particularly as you're spending a lot of time right now with admissions offices at major universities.

Well, as he talked about it, kind of this idea of retaking the SAT and how it really does help you in certain cases in terms of admissions. And, and I'm really interested. There's been a real big push on early decision at very selective colleges. Nick Anderson at the Washington Post did a big piece about a week or two ago on. On the huge growth. In many ways, early decision has become the new regular decision. Yeah. As students kind of understand that that's the way to kind of game the system.

Quick question, by the way, because it feels like early decision is one of these things that goes in fads. And is that true or is it actually largely, you know, increase? Like, is the line of early decision increasing or is it cyclical?

No, it's definitely growing.

Okay.

I haven't gone back enough in history to know if it's cyclical, but. But it's definitely growing. But it. What's happening now is that while the overall volume is growing, colleges still don't want to take more than like half their class early. So what's happening now is that the acceptance rate is dropping and most students think that that's an easier way to get in. And so what this requires, though, is essentially because most ED or EA deadlines are November 1st, is it really requires you to essentially complete your college search in your junior year, but to get your testing done in your junior year or at the very early summer.

Yeah, I was gonna say this, this is the Michael on the couch moment, but I remember taking the SAT junior year, and then I retook it in, I guess, September or October, whatever the earlies period was. And my application for Yale was due probably end of October. And I remember, I think I sent the scores, you know, as a, as a supplemental information for the committee or something.

Right. And one of the things that, you know, we were talking to Josh about is that there's a tremendous amount of growth in the teenage brain, you know, every month of high school. And so the fact that you're trying to, like, stuff, all this testing early on to get for these ED applications, I think ED really needs to be rethought. You know, it's incredibly unfair to low income students in particular, and also just the students who really don't know. I mean, there's Not a lot of research on retention, for example, of early decision students. Because one thing that I found interesting, I'm interviewing a lot of seniors for this book and many of them in mid October, so two weeks before the applications are due, said, I'm going ed somewhere, I just don't know where. And I said the purpose of ED is like it's your dream school, right? You really want to go there. And what they were really trying to do is just game the system to have. They're trying to figure out the best shot of getting in. The second thing that I found interesting is the use of SAT in admissions. So I'm kind of embedded in the admissions process at Emory, a large private. At Davidson, a small liberal arts college, and at the University of Washington, big public. And it's just really interesting in how they use standardized test scores. And the best predictor we know of college success is really grades great in high school curriculum and not test scores. Right. So as somebody said, you know, college is four years. We want to look at four years of work, not four hours of work, which the SAT is. And so you know, to me what the way I've seen the SAT used even in schools that are not opt. So these are schools that are not SAT optional is. It's kind of a check in, right? Like, well, we're kind of unsure of this student, what are his scores, right. But the best predictor is our grades. And so the question I have is that, you know, what is the future of standardized tests? You know, we've been seen a huge increase in test optional schools, University of Chicago being the most recent one last summer. So I'm kind of curious. We've had for ages, for really a generation, the two pillars of college admissions have been grades on one hand and test scores on the other. So more schools go and test optional. And I think there's a lot of questions about grades. Right. Grade inflation is up. And so if we don't assess students on those two things or we don't trust those two things, how should we assess them?

So I'm curious, do you see that behavior being different depending on the type of university? And what I mean by that is at an elite university, Davidson being a great example, small, where they, I think they spend a lot of time thinking about the qualitative nature of the application and so forth, compared to a large public, say, where they're getting, you know, lots and lots of volume and therefore they numerical assessment of someone may be more important. Does that differ or not? That's a misperception on my part, there

are more like cutoffs at bigger publics than at smaller privates where they have a lot more flexibility. But there are. Unfortunately, now I think it's not as easy. I was recently looking at the 1950. Michael Crowe, the president of ASU, gave me the 1950 admission standards for the University of California system.

Of course he did.

Of course he did. Right. But it lays it out in very stark terms, like what you need to get in. And if you do this, you get in. We can't do that anymore. Like, you know, even when we were applying, right, you could look at the ranges and you could say, okay, if I have these ranges, then I'm pretty, I'm pretty good at getting in. Now it is really like a lottery.

So. But, but some of the state, state universities are still like that and, but I guess they're, they're like that. On GPA though, if you're in the top 7% of your school, then you get automatic admission into UT Austin or something like that. Right. So it's not SAT based per se. And we've known that it's not predictive since, I mean, Tony Carnevale was doing research on this three decades ago.

Yeah, it goes back to that. So the other thing that Josh talked about, of course, was his research on the online master's degree in computer science at Georgia Tech and just found it fascinating, this idea that it really has first of all created a whole new market for students. And we've been talking a lot on segmenting the student market in new ways. And this is a growth area for colleges and universities. Let's stop thinking of students as this monolithic thing. But given your work on online education, what did you find interesting and replicable about what other universities could do?

Well, I'll say I love that it validated something that Gunner Councilman and I wrote a piece. When Georgia Tech initially announced that program, we said, finally, someone has crossed the Rubicon. We see the first actually disruptive program. Right. Because most of these programs don't actually classify as disruptive because they're not lowering the price point in the way that you would hope. And here was, it was, you know, order of magnitude less than the master's program on the Georgia Tech campus. And so what then I then loved was he went out and actually looked at who's taking it. And indeed they are non consumers. Like literally they would not enroll in a program were it not for this. And so it's like textbook what they're doing. And so I thought it Was besides validating, I thought it was fascinating. But I also took two other pieces from it. One, the amount of money that they had to put in to start this program, $4 million. That's a lot more than these, than these OPM led programs are putting into their programs. Georgia Tech really did this thoughtfully to build a sterling program, I think, and I think he's right. That's not something that can just be easily replicated. And then the second thing I thought was, gosh, what a shame that we should be building more high quality programs like this because. And this goes to the SAT question as well as this and maybe this is a good place to end with some thoughts on this question is the equity one. And so you know, in the satisfaction seems to be conferring an advantage on upper income students here. This seems to be conferring an advantage for certain institutions that are well endowed and I suspect or therefore means that a lot of these programs are going to be taken advantage of by people that already have degrees and maybe not out of the marketplace unless it's really thoughtfully designed. Just curious reflections on that.

And that's very true in the sat, right? The students who take it more often and who are ended up getting coached or people who have have access to that information. And as Josh said, the more often you take this test, the more often you know how it works and you end up doing better. And we know all the correlation research on the SAT shows higher scores are really highly correlated with income level. And Michael, I think this is really kind of the big issue. The big problem facing education in general in this generation is equity, right. And I see this more and more. I was here in Boston today before we recorded, visiting some of the students that I'm profiling as part of this book. And you see kind of the unevenness of the high school experience. You see this when I'm sitting in reading applications and you have students who go to schools that offer 50 APs and students who go to school offer one AP or two APs. You see counselor recommendations where the counselor has 30 students and then you see counselor recommendations when they have 500 students, right? And you just know kind of the advice and you talk to students about where they're getting advice from whose parents went to college and have been talking about college since day one. And then students whose parents didn't go to college. And as many of these students tell me, I can't go home and talk to mom and dad about this because

they have no idea.

And so I just think that there's always been this growing divide, but we know the demographics are equaling this huge surge of students coming down the pipeline who are of that lower socioeconomic status. And I just still don't think that higher education or education in general has adequately addressed this.

Yeah, and, you know, it points to something else which he was talking about his future directions for research and saying that he wants to dig in the actual ROI from these programs and micro credentials and the like. And oh, my goodness, I can't. You know, I'm not sure it would solve that problem today of the inequity, but I cannot believe we don't have the quality assurance mechanisms that are just out there connecting these databases of work, salary and education to answer these questions very cleanly. We just need. We need a more scalable quality assurance system in place so he doesn't have to sit there doing crazy research to answer what should be very basic questions that every student ought to have access to.

Yeah, and I just think it's great to have Josh on with us. As I said, we said at the end, he's kind of part of this new generation of researchers who are not only higher education researchers, who are not only economists, but also so really public facing. As a journalist and as a writer, I just love following people like that who really make their research much more accessible to the general public and really urge others to follow him and other researchers as well. So really thanks to Josh for joining us on today's episode and sharing his thoughts. And thanks to all of you for tuning in and be sure to rate us on itunes or wherever you get your podcasts. Until next time, I'm Jeff Salinger.