Supreme Court Justices Move Left Over Time (More than They Move Right)

My name is Mark Fisher and I'll be talking about some work I'm doing with a law professor I met online, Nicholas Georgiakopoulos. He was asking questions about a model that some political scientists developed to estimate the locations of Supreme Court justices on the left right spectrum and also see if they moved over time. He was not happy, that is to say, Nicholas was not happy with some of the assumptions they made in the model. And so we have modified them somewhat. Today I'll be talking about the model a little bit and I'll talk about how I use Mathematica to estimate the model. And finally I'll show some results about how the justices tend to move to the left over over time. All right. Thanks for coming. This is the title of the talk, as you can see, is Supreme Court Justices Move Left over Time More than they Move Right. And this is joint work with Nicholas Georgiakopoulos, who's down here. And I'll get started. So here's the genesis of this project. Sort of interesting. Nicholas asked a question at the Mathematica Stack Exchange Q and A community back in March about Markov Chain Monte Carlo estimation. And I answered saying two things, that he was confused and that it was not possible to straighten him out given the restrictions of the forum. And he thanked, he's very polite. He thanked me and provided his email just in case I changed my mind. And given his email, I was able to figure out who he was and I concluded that he probably was not a hopeless case. And so I contacted him and here we are. So the Internet is great sometimes. All right, so here's a very brief introduction and abstract. The paper that he was asking about Martin and Quinn in 2002, that's 20 years ago. Their model is still being used. It applies item response theory. I'll talk about very briefly about that. The Supreme Court votes in order to infer the location of justices on the simple left right spectrum, which of course is not the real world is much richer and more complicated than that. So this is a simplified model. And some of the things that Nicholas is especially interested in is what ways it might be feeling and how we might generalize that. But that's not going to be in the talk today. So we modify their model, adopting a non conjugate prior for some of the parameters. And we changed some other things around to the kinds of results that I'm going to show today that probably doesn't make that much difference. But for the kind of other things that we're interested in, it may make a substantial difference. And Although the finding that is mentioned in the title is not, is not new. I'm going to show some interesting ways to look at it. So. But if you're ever interested, the data are contained in a CSV file that's located at a public website@the supremecourtdatabase.org and here I want to show you this picture. Most people tend to think about the Supreme Court, as Martin and Quinn did, as a sequence of terms. The term is one year long. I think it overlaps two calendar years. And we have a database that contains 75 terms. The there's a legacy database that goes back to the 1700s. We might start looking at that, too. In any event, we're going to focus on compositions rather than terms. Now, a composition is just what it sounds like. If it's, you know, whoever's on the court, if somebody leaves or somebody comes, joins, the composition changes. And so what you can see here in this picture is how many terms a composition takes. So there's a total of 35 compositions over the 75 years. You can see some of them are very short. The compositions, in fact, some of them, I believe, are less than one term and may have only a very few votes associated with them. And some of them go on for more than 10 years or roughly 10 years. Some of the terms themselves, as you can see if you look carefully, have multiple compositions. So it is a bit of a different dividing up. Where are 39 justices in the database? And we allow each Justice's location to change with each composition for which the justice is a member. There are 307 justice locations that we're going to be interested in. What does the data look like? Given Justice J and Case K, the votes are coded as follows. The vote is a 1 if the justice voted with a majority and a 0 if the justice voted with the minority. This is almost no information at all. This doesn't say that the case is conservative or liberal or anything like that. It just says you voted with the majority or the minority. There's a handful of evenly split cases. We don't look at them. And we also don't look at unanimous cases because they're not really going to tell us very much about what we want to know. There are about 40,000 votes on about four and a half thousand cases over the 75 years, or 35 compositions. So what, what's, what's the model here? What's this item response theory? And I'm not going to give you the background of item response theory, but this is what it boils down to. So the vote, which probably should have a T in it right here because it's the. No, no, that's the vote for justice j on case K, which may happen to be in composition T is a Bernoulli variable. As you saw, it's either 0 or 1. And so there's a probability. The model says there's a probability that the vote is one. And the item response theory says that probability depends can be modeled as this function here. So that capital phi is the cumulative distribution function for the standard normal distribution. So it's like an S curve and it either slopes up to the. That's your right or it slopes up to the left, depending upon the sign of what's in the argument there. So theta is the location of the justice at time t. Theta jt is the location of a justice at time t. Gamma K is a case parameter that's telling us something about the case. So a case is completely summarized by a simple curve like this. So it's another complete, very, very simple approach. The amazing thing is how much you can get out of this. The theta, as I said, is the location of a justice J at time t. And then each case has these two numbers, gamma and delta. And if you look inside the CDF there, where the parameters show up, there's a couple of things you can see. If you change the sign on all the parameters, nothing changes. What that means is this model knows nothing about absolute left or right. We have to tell it, and we don't have any. We're not using any information from the cases that says that. So we have to ultimately say, well, Justice Thomas is on the right or Justice Douglas is on the left. Once we pin that down, every, everything that that's been determined. But we have to say that because it could just be, could just turn, could just pick the other way. The other thing is, if you look at the difference between theta and gamma, if you add the same amount to both of those, that doesn't change anything either. So that, that is reflecting the fact that there isn't really any absolute zero here. What we can talk about is where justices are located relative to each other and how they change over time. And. But we just basically start things at zero and center them on zero. So what do we need in order to make progress? Well, we have 307 justice locations and 9,306 case parameters. We've taken a Bayesian, as Martin and Quinn did, we're taking a Bayesian approach to estimation. And so we need to come up with priors for these roughly 3,600 parameters. 9,600 parameters. OK, so very briefly here is Bayes rule. So let's see. Do I have a pointer? Okay, so that is the collection of all locations. So that's all 307 locations. That's Theta. Eta is the collection of all the parameters. What were they? 9,000 parameters of all the case parameters. Excuse me. And then phi is all of the unknowns, the locations and the case parameters and the data, all of the observations are collected into this object Y. And then we have a joint distribution between the data and the unknowns. And that joint distribution can be factored two ways into conditional and marginal. And then if we can then solve those two different factorizations to get Bayes rule, which says that the distribution of our unknowns given the data, which is what we observe, is the expression on the right hand side and these different parts have standard names, what would have been the sampling distribution? But we're not sampling anymore. The observations are given is the likelihood, phi is the unknown thing in there. And then P of phi is the priority and the posterior is the thing we really care about. So for us, estimating a model is coming up with this posterior distribution for the unknowns, all of them. And then maybe we'll look at some means and variances and stuff like that. But we really want to get the entire distribution so we can characterize what we know and what we don't know about these things. As I say down here, once we observe the data, they are fixed. And the thing in the denominator P of Y is just a number. Sometimes it's an important number, but for us, for our purposes it's not. And so Bayes rule is often written as this proportionality where you just say I need the likelihood, I need the prior, and I can make progress. Well, our fee, remember, is made up of two different kinds of parameters, right? Two different kinds of unknowns. There is theta, which are all the locations, and there is eta, which are all the case parameters. Now there's a remarkable theorem that says that any joint distribution is completely characterized by the full by the complete set of full conditional distributions. Well, in our case that's two. I mean, there's a lot of vectors and stuff in here, but we're just treating these things as individual objects. There's two of them. There's the conditional distribution of the locations given the case parameters, and there's the corresponding conditional distribution of the case parameters given the locations. What that means is in terms of estimating the model, is we can alternate back and forth between these two. We can say, well, if I knew the case parameters, then I could sample the locations. And if I knew the locations, then I could sample the case parameters. And the Gibbs sampler is a Markov chain, which is constructed doing exactly that. And you're guaranteed to get the correct probability, joint probability distribution. Using that, you're not going to get IID draws, but you are going to get the correct distribution. If you do enough of them, you got to do a bunch of them. So this is an example of Markov chain Monte Carlo. Now we're going to get a further decomposition. Now that we recognize. Okay, let's look at the case parameters themselves. It turns out the way they structure the model and way we structure the model, each of these case parameters can be taken independently, conditioned on the votes. And so we can estimate or we can make a draw from every one of those separately without reference to the others. Once we assume we're seeing the locations, which of course will change from iteration to iteration. And similarly, we can do exactly the same thing for the location. So this decomposes the model in a way that it makes everything very much simpler. And we just have a whole bunch of relatively simple draws for our MCMC and Gibbs. I'll just skip over this. Okay, one, one point. When, when, when, when Martin and Quinn made this model 20 years ago, computing was a lot more expensive. So I don't know how you calculate Moore's Law. Is it every two years? Sort of. Things get twice as powerful for the same dollar amount, for the same real dollar amount, or is it every 18 months? Well, you know, you get about a factor of 1,000 or 10,000 in computing power, depending upon how your view of Moore's Law is, which means that we can do a lot more in the same amount of time than they could do. And so there were choices that they made in their model in order to get expressions that would allow them to make draws in a very straightforward, simple manner. We're much freer to be able to choose priors for the case parameters and for the locations. It's mostly the case parameters where we've changed this to make priors that don't produce conjugate that are not conjugate, but might be more interesting, they might be more useful, they might be more meaningful. We don't get the simple expressions, so our calculations take a lot longer than they would otherwise. But we can afford to do that because computation has gotten so much cheaper. Just very briefly, here was their parameterization, in which the case Parameters and the location go into the CDF that determines the probability for the Bernoulli for the vote, which way the justice is going to vote. This is linear, it's that linearity. And they can put normal distributions on alpha and beta and then that feeds into other parts of the model. That makes things about as efficient and simple as possible. But that doesn't make it right or that doesn't make it the best thing. I'm not criticizing them for doing that back then. On the other hand, the model is being estimated. It's a great model and it's being estimated every year. My understanding is they're still doing the same thing. Well they could do other things and that's part of what we're doing. So in our parameterization it's somewhat different. Now the prior for the case parameters turns out to can be important for the following reason. For every case, how many observations do we have about Justices locations? Well there's at most nine Justices voting, sometimes there's fewer. So we don't have for any given case a lot of information. And yet it's the cases themselves. Now there's a lot of cases, it's the cases themselves that are going to give us information about the locations. And so the fact that there's this feedback means that experimenting with different priors or adopting different priors could make a substantial difference for some aspects of the posterior distribution. So as I said, they put a normal distribution on beta. Now if you look at the expression up here for beta, they have a normal distribution that's centered on zero. Well if beta happened to be zero, now the prior isn't going to make it zero, but it encourages it to be zero, then in fact location doesn't matter. Everybody has the same probability. And so we're not going to get much information back that way. So my idea was no, what we're going to do is we're going to, we're going to flip beta over, we're going to put a normal distribution on that and we're going to tell this thing that it's not going to be zero unless it really tries hard. So here, here's an example. The blue line is a normal distribution and the other colored line, I always wonder what that is. Is that brown? Is that mustard? What is that? That other colored line is the distribution for 1 over x if X had a normal distribution. So our distribution for delta is normal and hence our implied distribution for beta is that other thing. And look, it says you're not going to be zero, you have to choose. You have to make a commitment here. Don't be zero. Now I will tell you that in the results sometimes you just can't tell. And so it jumps back and forth between those two peaks. Okay, so it says I want to be zero, but I'm not allowed to be zero. So I'm going to do the best I can. But this does make a difference and we think it makes a useful difference. I'm not going to spend any more time on that right now, I think. Okay, so here is the posterior distribution for what we get for our parameterization. So gamma is on the horizontal and delta is on the vertical. And you will note that these are not independent in the posterior distribution. There is strong dependence and in fact our prior encourages that. Even if you didn't have a prior that encouraged that, you would still find that. But we have a. So the prior for delta isn't just a prior, for beta isn't just one over a normal. It actually involves saying, well, if delta, if delta is positive, then Gamma is going to be normal with a certain mean. But if delta is negative, Gamma is going to be normal with a different mean. Again, this is in the prior and the data can overwhelm this. But this is not a bad, this isn't all that far from the prior. Now one thing you could ask is, but what do Alpha and beta look like if you just calculate them from this? But these are not all the draws, by the way. As it says, these are the posterior means. The draws will spread out more and be a fuzzier cloud. Here are the alphas and the betas. Alpha on the horizontal, beta on the vertical. And I love the scaled, the log, scaled, scaled log scaling function because there's so many times when I wanted to do that and I didn't know what to do myself. And so you can see these things are all over the place. And it just shows that the parameterization makes a difference in the prior makes a difference, at least at this level. All right, what about the locations? Well, remember, the locations are going to change from time to time over time. Martin and Quinn set everything up so that basically they could use what's called the Kalman filter because they had normality and, and linearity. And so they could run, they could do a Kalman filter forward through time as they're looking term by, they're looking term by term by term for a given justice, holding everything else fixed and then sampling on the way back. So it's a forward backward. And I'll talk about this a little bit More. It's a forward backward algorithm for making draws for a given Justice's locations jointly from all of the Justices data and all of the case parameters that they voted on. I didn't have the ability to do that. And there are all kinds of ways that you can use non Kalman filters like particle filters. And one of these things is called the unscented filter. I don't know if you ever heard of that. And the question why do you call it the unscented filter? Because it doesn't stink. So there's all kinds of crazy things out there. I don't want to use that kind of stuff. I want to use something that's sort of. More, it's closer to. Well, here's what I wanted to do. So what I did was I discretized the world into this range that goes from minus 4 to 3. I've already estimated the model a number of times. So I know using the roughly speaking what range is going to capture everything we need. And I got a step size of 0.01. So there's 701 points on the grid. And then I'm going to fill out this. I'm going to say the initial position on the grid is a discretized normal. And then I'm going to have a transition function which is a matrix. And the matrix transition function, the transition matrix is again going to be discretized normal. And so it's just sort of capturing the same kind of a flavor. And that's what shows up in this expression here. Basically theta range is that 701. And so here's some code that ends up creating a matrix we need. We end up with a setting that looks like a hidden Markov process. And hidden Markov process is a symbol that's built into Mathematica. But we don't use it. Why don't we use it? Two reasons we don't use it. Okay, the first reason is that on this page or is that on the next page? It's on this page, I guess, yeah. Two reasons we don't use it. The first is that if you look at hidden Markov process and you give it some observations, I think they're called emissions in Mathematica. And you say what states would be associated with this? It returns one sequence that most likely. Well, we're doing Bayesian stuff where we have to, we have to draw samples and so that's no good. I have to be able to draw a sample. So it's easy to fix that. Just add something to it and Allow it to turn either of the two forms, one of which has a matrix and the other which has a list of distributions, and put it into the forward backward algorithm and get samples. That's no problem. The problem is that there's still more structure built into or required by hidden Markov process that we don't have. Remember that some of these compositions have lots of years and hence lots of votes in them, and some of them don't have so many and our emissions, therefore they're not time homogeneous. Because we may have only a few votes in one composition for a given justice and hundreds of votes in another, we have to construct the likelihood, which is what's going into this, from that. And there isn't a simple formula to do it. And so we do that on the fly. I mean, we do that for our case and then we get this wonderful. I love this here. I mean, we get this one. We have this wonderful forward backward algorithm that takes those inputs. L is the matrix of likelihoods, which is based on the votes of the justice for. It has many rows, one for each composition that the justice is in and many columns, one for each of the grid points. And Q is the transition probabilities. Remember I said we get those by discretizing a normal distribution and G0. I said we're going to initialize everything with a Gaussian distribution as well. Those are the inputs. And then we have, look at this. It's two fold lists. I just love this. There's a fold list going forward and there's a fold list going backward. And in both cases I end up having to reverse it, because in fact I have to reverse it to feed it first of all, to feed it into the backward part, and then I have to reverse it so that I get the right thing out at the end. And so basically what this thing does is it rolls forward, doing matrix calculations, calculating filtered probabilities, just like the Kalman filter does all the way up to the end of time. And then you take a sample from the last one, and given that draw, you can then calculate going backward. And using the Markov properties, you can calculate going backward what the probabilities are given that one draw, given where it is on the grid, so you don't have to do this huge, huge, huge, major calculation which would explode and become impossible. And that's what this does, and it's very straightforward. Now, one of the things that I discovered using this was that in fact there was a bug in compiler with regard to this. And so that's why there is this use. That's why there, this, this width up here is being used. Because what was happening was when, when the thing that was going here to start the whole thing off was being. It was. It's random. It was being evaluated. And then when it stuck it in here, it evaluated a second time and gave me a different result. And so it was somebody at Mathematica Stack Exchange who said, hey, this is what you should be doing. All right, so here's some results. This is not easy to read, and it's not what we're going to focus on. But these are the justices, and these are the compositions, 35 of them. And these are the means of their locations over time. So Justice Douglas down here, getting more and more liberal. That's to the left, negative numbers, and then getting a little bit less liberal and then getting more and more liberal. We've got Justice Thomas up here. I'm pretty sure that's Thomas. And anyway, you could go on and look through these, and if we had time and inclination, we would look at each one of them separately, et cetera. Now, let's put this on a term by term basis. So now some of those get shortened and some of those get lengthened. And this is what it looks like if you just look at it in natural time. Same sort of a story. I think this one right here, this purple one is Justice Rehnquist. Justice Rehnquist was about as conservative as you can get until he became Chief Justice. And then he got more moderate. He didn't get liberal, he got more moderate. All right, here's some. Just a sense for uncertainty, okay? I'm not going to go to a large, to a large extent into this, but this is a sense for uncertainty. You can just take a bunch of draws and say, what do they look like? Are they really separate from each other? Are they really going up? Are they really going down? That kind of thing. All right, and now we're going to look at first and last compositions and get at what the title of this thing is about, which is to show, give some sense for the probability that justices are moving left or moving right. And I'm not going to. So what I say here is we're going to look at the joint distribution two ways. One is a discrete way, one is a continuous way, and we'll just go straight to the numbers. So in this table, we can say, what's the probability? U. U means injustice. Move to the left, okay? And the answer is 63%. If you start right, what's the probability? You move to the left? Well, if you start right, it's only 55%. But that's bigger than 50%. If you start left, what's the probability you move left? Well, that's 80. Roughly 80%. If you end right, what's the probability you moved left? Well, it's still 42%. If you end left, what's the probability you move left? Well, 86%. That's very strong. And if you end left, what's the probability you started right? 30%. But what if you end right, what's the probability you started left? 1%. Okay, so that tells you something. Here's the joint distribution in continuous terms for what we were just looking at. It's probably easier to see here in this contour plot. And you can see this is. There's a. There's a big mass right here. And on the conservative side, this is the dotted. The 45 degree line is if you didn't move. And I've been told to stop. And so I will stop as I show you this here, which I should have gotten to sooner and I apologize. These are conditional distributions that say if you started somewhere, what's the probability you would end up. And the point is, if you start on the left, you end up farther to the left. And if you start on the right, you tend to stay around there. Thank you.