#43 Election Forecasting and Polling

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Andrew Gelman about statistics, data science, polling and election forecasting. Andy is a professor of statistics and political science and director of the Applied Statistics center at Columbia University. And this week we'll be talking the ins and outs of general polling and election forecasting. The biggest challenges, engaging public opinion, the ever present challenge of getting representative samples in order to model the world, and the types of corrections statisticians can and do perform. Chatting with Andy was an absolute delight and I cannot wait to share it with you. I'm Hugo Baun Anderson, a Data Scientist at DataCamp, and this is Dataframe. Welcome to Dataframe, a weekly Data Camp.

Podcast exploring what data science looks like.

On the ground for working data scientists.

And what problems it can solve.

I'm your host, Hugo Bown Anderson.

You can follow me on Twitter ugobaun and Data Camp at Data Camp. You can find all our episodes and show notes@datacamp.com Community Podcast hi there Andy, and welcome to Data Framed.

Hello.

Such a pleasure to have you on the show and I'm really excited to have you here today to talk about polling and election forecasting. But before that, I'd like to find out a bit about you, just to set the scene. So my first question is, what are you known for in the data community?

What is the data community?

The data community, I think, is the rough, broad collection of people working with data analytic techniques, working with data science, and working with large and messy data sets.

These days I'm probably best known as.

One of the authors of the book Bayesian Data Analysis, which came out in 1995, but then we've had two more editions since then.

So that was a book.

I like to think of it as the first applied Bayesian statistics book.

So a lot of people who've gotten.

Into Bayesian statistics have gone there through our book or use our book as a reference.

Great. And maybe you can tell us a bit more about Bayesian statistics in general, just by way of introduction. And I suppose there are two types of statistics in general that we talk about, which are Bayesian statistics and frequentist statistics, right?

So in Bayesian statistics, all of your unknowns, all of your parameters, unknown parameters and predictions, are associated with a probability distribution. So the way you solve a problem using Bayesian inference is you put all of your knowns and all of your unknowns into a joint probability distribution and then use the laws of probability to make statements about the unknowns given the knowns.

And so you've actually done a lot of work on implementing a lot of Bayesian techniques in a language called Stan.

Right.

In fact, a language in which, as you mentioned, probability distributions are kind of the core objects of Bayesian statistics. I suppose distributions are first class citizens in Stan and other what are known as probabilistic programming languages, Right?

Right, exactly. So I can give you a simple example.

Suppose you're doing an educational innovation and you want to look at students test scores after the intervention. So you start with basic statistical ideas.

You fit a linear regression model, say.

Predicting the test score given their pretest score and given an indicator for whether they got the treatment or the control. So that's regression, that's not Bayesian yet, it's just statistical modeling.

And it can become more or less.

Difficult, it can become nonlinear. You can control for more predictors, not just your pretest, but all sorts of student characteristics. There's a million things you can do. What makes it Bayesian is that this regression model has parameters like the effect of the treatment, how much the post test is predictable from the pre test. There's parameters for how big your variance is, shapes of distributions, whatever. All of those parameters are assigned a probability distribution called a prior distribution.

So you put that all in Stan.

Along with your data and, and then it gives you a posterior distribution which represents your uncertainty about the parameters after seeing the data.

And so Bayesian data analysis and Bayesian inference, I think historically we've seen them to be incredibly powerful, but maybe not haven't been adopted as widely as Bayesians would have liked. And I think a lot of learners, a lot of people learning data science and statistical inference may find Bayesian data analysis even a bit scary. And firstly, is this right? And secondly, why is that the case and how can we correct that?

Well, in Bayesian statistics you kind of make a deal with the devil. You assume a probability model, so you make a big assumption and having done that, you can make predictions about Just about anything.

So I think maybe it's a little.

Scary in some way because it's so powerful and so easy to use. It's like those 3D printers, right? People are afraid of them because they can print anything. So in Bayesian statistics, even if you have weak data, you can get inferences, and the inferences then become driven by your prior distribution. There's a saying we have in Bayesian statistics. With great power comes great responsibility. And what that means is that in Bayesian inference, it's very important that you check the fit of your model and check the reasonableness of your model. So. So in that sense, there's kind of two approaches to statistics. One approach is to make very minimal assumptions, and the other is to make maximal assumptions. The Bayesian approach is really you make maximal assumptions. What I like to say is you create a paper trail going from your assumptions to your conclusions. Then if your conclusions don't make sense, you look at what was wrong with your assumptions, and what was wrong might be your model for your data. Maybe your sampling was biased and you didn't recognize that. But whatever it is, somewhere you need to go back and forth. You need to communicate between your assumptions and your conclusions. And I mean, a lot of people would rather work without assumptions, and sometimes you can. We can talk about examples, but basically, if you have a clean problem and good data, then you don't need to work with a lot of assumptions. Well, except for the assumption that you have good data. As the data quality becomes worse, as your questions become more difficult to answer, you need to put more assumptions in, then Bayesian inference becomes more useful.

Absolutely. And one of the great things that you mentioned there was the ability to check your model after the fact. And we have enough computational power to do that these days. Right. So, for example, once we have our model, we can simulate what the data would actually look like and compare that to the data we actually saw.

Exactly. We call that posterior predictive checking. Now, people have been doing this for a long time. They're just not under that name. There was a book from the 1950s by the statistician Frederick Mosteller, where they were analyzing it was data from an experiment, was called a stochastic learning experiment. They were actually giving electric shocks to dogs in cages and seeing how long it took for the dogs to figure out that the shock was coming. So they had this probabilistic model, and then after fitting the model, they simulated fake data, and they compared the fake data to the real data. And in the 1970s, the statistician Brian Ripley, who was working on spatial statistics and since has become very famous for his involvement with R. Brian Ripley was fitting spatial models and again did the same thing. He had a model that seemed kind of reasonable. He simulated replicated data from the model, and it didn't look like the real data on that that inspired him to expand his model. So it was examples like that that motivated us to formalize this idea of model checking. And I think people have always checked their model, but there's been a sense in which it's been like, outside the system. It's not that people are embarrassed to check their model, but it's almost like people think like, it's like, well, I'm a good person, I'm a good citizen, so I check my model. And it hadn't been formally encompassed into statistics. And in the Bayesian framework, you can do that. You can put model checking right in the middle of the process and not feel that it's some external thing you're doing.

I'm glad you mentioned that, because that was my next point, that it is actually baked into the Bayesian workflow, the idea of model checking.

Yes.

So this was Bayesian data analysis. Are there any other things that you're known for in the data community?

I'd like to say that I'm known for statistical graphics because in the. In the early 2000s, I did a lot of work trying to integrate statistical graphics with statistical analytics. So traditionally, there's this idea that exploratory data analysis is looking at your data and finding interesting patterns. And confirmatory data analysis is like crunching the numbers and getting your P values. And exploratory data analysis, again, was kind of on the outside of statistics. Its proponents would often say, for example, forget all this silly modeling stuff. Let's just jump to looking at the data. But what's interesting is, if you think carefully, exploratory data analysis is finding the unexpected. So to say I'm finding the unexpected means that's relative to the expected. And in fact, exploratory analysis is most powerful when it's tied to models. So I think of exploratory data analysis and statistical graphics and learning new things from data from visualizations.

It's actually fitting in very well with.

Bayesian inference and formal statistical modeling because you fit the model. The better your model is, the more you learn from its falsification. So, like way back, Copernicus had the model that the planets were going in circular orbits around the sun. And it's easy to falsify that, but then Kepler moved to the elliptical orbits of falsifying, that became more interesting and so forth. So every time we have a model that motivates more sophisticated graphics, which allows us to learn more.

So how did you get into data science and statistics?

Originally? I always was good at math, ever since I was a baby. And then I've written about this, actually. But anyway, when I was in high school, I did the Math Olympiad training program. I found that there were people better at that than I was. We had very naive view back then, so we didn't know about applied math, we just knew about this thing called math. And we thought ability was kind of unidimensional. But anyway, I went to college and studied physics and math and I didn't want to be a pure theoretician. I just felt I wasn't good enough to make useful contributions in that way. I. I first took a probability class because it was in the evening, like it fit my schedule. So I took probability and stochastic processes and then took statistics. And I really like that in statistics there is kind of a continuous connection between everything that I care about. So there's a connection between things that I can do, like mathematics and also things like politics, public health, economics, sociology, like all those things. There's kind of a continuous thread from these qualitative thoughts about what's going on in our country, what's going on with the world. How do people learn all sorts of things like that through qualitative thinking, statistical modeling, mathematical analysis, programming, all those things. So it was really perfect for me. I sometimes think that statistics should be called mathematical engineering. Like they have electrical engineering and mechanical engineering and statistics is mathematical engineering.

I like that. And something you hinted at in there, or spoke to directly is that it is this marriage of your aptitude and mathematical skills, but also your serious deep interest in the political and social science.

Yeah, in college I minored in political science and so I found that very interesting. I mean, political science is a funny field because you don't make progress in the same way as you do in a technical field. You can say, technically we can do all sorts of things that Gallus couldn't do or whatever. I'm sure he could figure it out when he saw it. But, you know, we just know stuff they didn't know in politics. Like what do we know that Hobbes didn't know? Well, it's hard to say, like a lot of specific things, like the size of the incumbency advantage and so forth, but it's, you know, it's a little bit different, right? Like it's more like something like architecture. Like we have buildings now, but it's. You're just, you're building things that serve current purposes. Then maybe the principle that technology changes, but the general principles aren't changing.

We'll jump right back into our interview with Andy after a short segment. Now it's time for another installment of Blog Post of the Week. I'm here with Emily Robinson, a data scientist on the growth team here at Datacamp. Hi Emily.

Hi Hugo. Thanks for having me on again. This time I'd like to follow up on our previous segment on the Data Ethics series by Mike Lukides, Hilary Mason and D.J. patel and discuss their third article, the five Cs.

Great, Emily. So what are these five Cs?

They are consent, clarity, consistency, control, and consequences. This article is about how these can be used as a framework for implementing their golden rule for data to treat others data as you would have others treat your data. Let's start with consent. This is about needing agreement from the users about what data is being collected and how it will be used. The authors discuss a problem with data services contracts. Though they're often binary, either you accept the terms or you don't get access. They're non negotiable and unfortunately, data is frequently collected, used and sold without consent.

And what's the next circumstance?

The next is clarity. Clarity is closely related to consent because you can't consent to something you don't understand. For example, while Twitter users generally know that their tweets are public, they often don't understand that they can be collected and used for research or even sold. Consistency and trust are another component. You need to be consistent to be trusted. An organization can break trust unintentionally when there's a data breach after they failed to safeguard customer data. In the case of Facebook and Cambridge Analytica, Facebook said at first it wasn't a data breach because the data wasn't stolen, but the public's perception was different. Facebook then became unpredictable because users didn't know how they would handle different situations.

So then there was control and transparency, right?

Right. Users generally don't have control over their data. They're given the all or nothing choice described above in consenting to use the data service. So it's often impossible to reduce the amount of data collected. It's also difficult to have it deleted later and to verify that it was in fact deleted. Finally, the last C is consequences. As the authors put it, often unknown. Unknowns are unknown because we don't want to know. So even though there are unforeseen consequences and risks that can't ever be eliminated completely, many could be seen if people tried harder. They're not advocating for never sharing data. The author cites several examples, including the Department of Transportation releasing data about traffic fatalities where data was responsibly, publicly released. But for it to be done safely, there needs to be lots of careful planning.

Thanks, Emily. So what should our final takeaway be?

The authors advocate for the five C's to be part of every organization's culture. Product and design reviews should go over it, and there should be checklists to make sure they're always being considered, even for MVPs or minimum viable products.

Great.

Thanks, Emily, for another great installment of.

Blog Post of the week.

Thanks, Hugo.

After that interlude, it's time to jump back into our chat with Andy Gelman.

Before we get into polling and election forecasting, I just want to speak more generally to data science and statistics. I'm just wondering, Moving forward, it's 2018, right? Moving forward from now, what do you think the biggest challenges facing data science and statistics as disciplines are?

Well, speaking generically, I think there are three challenges of statistical inference. The first is generalizing from sample to population, and that's a problem that's associated with survey sampling, but actually arises in nearly every application of statistical inference. You never like, even if you have like, people sometimes say, well, wait, I have Data on the 50 states. That's the population. We're not going to have a 51st state anytime soon. And then even then, I would respond, okay, you have data from the 50 states last year or the last 10 years. What you're interested in is the 50 states next year, right? So there's always some generalization that's involved. So ideas of statistical sampling always come up. Now, the second fundamental challenge of statistics is generalizing from the control group to the treatment group. There's much of the time we're interested in the effect of some treatment or intervention, and obviously things like drugs or educational interventions or business decisions, but all sorts of social science things. Whenever you ask, why is something happening? You're implicitly asking, what would happen if I change something? And with rare exceptions, we don't have a matched control and treatment group. Typically, the people who you can do something to are different from the people who didn't get the treatment, and so some adjustment needs to be made. And then the third is generalizing from observed measurements to the underlying constructs of interest. So that's most obvious in something like educational testing.

You Want to know ability, but what.

You get is test score. So we spend a lot of time designing instruments, designing survey questions, lab measurements, what those people at Theranos, those fraudulent blood testing people, that was all about measurements. So when you talk about challenges, I think those are all, those are the old challenges and they remain the new challenges. Big data tend to be messy data. So it's not a random sample, it's convenient sample, it's an opt in sample. You don't have control and treatment group. People choose their own decisions of what to do. And often you don't have careful measurements of what you care about. You often just have data that are available from another source, which you're trying to adapt. And so for that reason, if you want to get good predictions and sensible answers and learn, you need to adjust for differences between sample and population, you need to adjust your for differences between control and treatment group, and you need to model the connection between what you care about and what your measurement is. Now all that can take a lot of modeling. So typically we say that you either can get, you either get like good data or good model, or you have to have a mixture of both. You have to do a little bit of data, a little bit of work. You have to work on data collection, you have to also work on the modeling. So if you have big data and you need big model, then that's going to require a lot of computation and that's going to be expensive. And so you need algorithms for fitting models, approximately fitting models. We have some sort of good things in our corner. Like for example, as you get a lot of data, often your inferences will become more stable. They won't necessarily converge to the right answer, but things might look more normally distributed. That's from the Central limit theorem. So that suggests that certain statistical methods, certain approximations might work well when you have a lot of data, which is good because when you have a lot of data, that's when you need the approximations more. So there's a lot of things like that, like moving between there I've been sort of moving between applications and research agendas. But the research is to fit these big models and to understand them. And that's continually going to be a challenge.

So those are all really important points that we'll actually see focused even more through the lens of holding an election forecasting. But before we get there, this idea of statistical inference and statistical modeling, I'm wondering what it takes to be able to be part of that conversation. I suppose my question is, as humans, we don't necessarily have good statistical intuition. And I'm wondering how you, as an educator and statistician, would like to see statistical and data literacy change general for a general population.

There's different ways of looking at it. Some of this is procedural. So if there is an expectation that.

When you have an analysis, you put your data on GitHub and you put your analysis on GitHub and it's all replicable, that'll change.

I think that alone will help.

That won't make people's analyses better, but it'll make it easier for people to see what went wrong.

It's surprisingly difficult to get people to.

Say or write exactly what they did.

I find this with students, but even like I've been in consulting settings where.

Maybe there's an expert on the other side and they do an analysis and they write up their analysis and you can't understand what they did. They'll photocopy three pages from a textbook and say, we did this. And they don't say where their data.

Come from or anything.

I've come to realize that a lot.

Of people don't even know what they did.

So people don't have a workflow. They just have a bunch of numbers.

And they start screwing around with the numbers and putting calculations in different places on their spreadsheet. And then at the end they pull a number out and write it down and type it into their report. There are those famous examples like that Reinhart and Rogoff Excel error from that Econ paper from a few years ago.

But lots of published journal articles where.

Not only do the results not replicate, but people have gone back to the articles and found that the numbers in the paper aren't even consistent with themselves. For example, they'll say there is a certain number of cases and then they'll have a percentage, and the percentage doesn't correspond to any ratio with that denominator. Or they have the estimates and the standard errors and the Z scores, but they don't correspond to the same thing.

And I'm just starting to realize people.

Just don't have a workflow at all. So requiring a workflow would help when.

It comes to understanding.

There's something you might have heard when you were a kid, which is if you have difficulty with a math problem, put dollar sign in front of it, and then it's much harder to be off by orders of magnitude.

And psychologists such as Gerrit Gigerenzer and.

Others have put a lot of work into understanding our cognitive illusions and how we can fix those Problems.

And one idea is to move away.

From probability and move towards frequencies. So there are these classic probability problems like you have. There's a disease, and 1% of the people have a disease. And you do a test, and the test for the disease has 98% accuracy. Somebody tests positive, what's the chance that.

They have the disease?

And it's very hard to do that in your head. But what you can do is say, imagine you have an auditorium with a thousand people in it. Well, I just told you 1% of the people have the disease. So picture 10 people in the front row of the auditorium.

They're the people with the disease.

The other 990 don't. Now we're going to do a test with 90. Did I say 90% accuracy?

98.

No. 98% accuracy.

Well, that's tough because you have to.

Do 98% of the 10 people. So then you need higher numbers. So let me rephrase that and say it has 90% accuracy. Just to keep the algebra simple. The test has 90% accuracy. So then you look at the nine people in the front row. Well, nine of them test positive, and one's going to test negative. And you look at the 990 people otherwise. And out of them, 99 are.

990.

90. 99 are going to test positive by accident.

That's 10%.

And then the others are going to be negative.

Then you take all the people who.

Test positive, you have them raise their hand, and you see that we have nine sick people who tested positive and 99 healthy people who tested positive. So most of the people who tested positive were healthy.

So the amazing thing is I could do that all by speaking in my head. And I couldn't solve the first problem in my head. Now you could say, well, but I.

Had to sort of screw around with the numbers because 98% didn't work.

But that's kind of the point that if you have a 1% disease and the test has 98% accuracy, you really.

Can'T solve the problem by thinking of a thousand people. You need a larger population.

So we could think of a city with a million people, and now, well, 1%. So 10,000 people have the disease. And I'm purposely talking this through to.

Show that you can do it.

10,000 people have the disease and 990,000 don't. You could write this down, but you could try it in your head. Then of those 10,000 with the disease, 98%.

So that's going to be 200.

So, like, I could change the Numbers around a little.

I could do it in different ways.

But the point is that like actually.

Having that denominator makes it easier to visualize. It makes all the numbers make more sense.

So Gigerenzer's argument is that the denominator.

Really is always there and the denominator actually matters.

There's a difference between something that happens.

10% of the time to 10 people compared to something that happens 10% of the Time to 10,000 people. It's a different phenomenon.

It would be a little bit like looking at. I mean probability theory is great, but. So the answer is there are ways.

Of understanding probability better by thinking in terms of frequencies.

And this is something we've actually seen in election forecasting as well. So this will prove a nice segue. I know at least 538 and Nate Silver's House model, they like to think in terms of. They won't say, we predict that the democrats have a 75% chance of getting the house. They'll say a 3 in 4 chance because they feel that heuristically that helps people formalize it a bit better. They'll know one out of four times the Republicans will get it three out of four the Democrats will. And then you can even think in those terms. What does 1 in 4 mean? That's the probability of the frequency of getting two heads in a row, which isn't really like which. You wouldn't be surprised if that happened, right?

Oh, sure.

This happened before. I can tell you a story about.

Nate, but first.

Before the 2016 election, someone said, well, what about this forecast? It's like there was some model that gave Clinton a 90% chance of winning. They said, well, 90%, how do you think about that?

And I said, well, there's a presidential election every four years. 90% is 40.

So 10% means something would happen roughly.

Once every 10 elections or every 40 years.

Well, I remember about 40 years ago.

In the 1980 election that it was supposed to be very close and then Reagan won BY I think 7 percentage points.

So it was a big surprise, much.

Different than what was expected. So yeah, I think it could happen. Sure. Like there could be.

Now actually, Clinton did very close to.

What she was pulled. She was supposed to get like 52% of the two party vote and she got 51%. So the polls are better now than they. In some ways the forecasts are better now than they were in 1980.

But that's how I calibrate one in 10. And as a political scientist, I often.

Say I don't like 95% intervals, there's a 95% interval supposed to be correct. 95, you know, 19 times out of.

20 for 20 presidential elections, that takes 80 years.

I'm not really. I think it's kind of ridiculous to try to make a statement that would be valid over 80 year period because politics changes over 80 years.

Now my story about Nate was in 2000, I think it was in 2012. He was going around like he said, oh, Obama has a 65.8% chance of reelection. The next week he'd say it was 63.2%, then it was 67.1% and it would jump around. It was meaningless. You can say he has a 60% chance, but to say a 65.1%, you can do a little bit of mathematics. What you can do is say, let's.

Predict his share of the vote.

Let's suppose he was predicted to get.

Something like 52 or 53% of the.

Vote and there's some uncertainty.

You have a little bell shaped curve.

And if it's less than 50%, let's.

Temporarily forget about the electoral college for a minute. That's not really the concern here.

The point is if his electoral votes.

Are predicted to be less than 50%, then he would lose.

Otherwise he'd win.

Now to say, suppose you say the probability is 65.8%. That's going to correspond to a certain.

Bell shaped curve saying he's with his like expected number of votes and uncertainty.

Now it turns out if you wanted to shift that from 65% to say 66%, that would correspond to shifting his forecast share of the vote from something like, I don't remember the exact numbers, but something like 52% to 52.01%, something like trivial like that. So it's a meaningless number. It would be like saying Steph Curry.

Is 6ft 3.81724 inches tall.

And so I sort of got on Nate's case and I said, look, you.

Know, I understand Nate.

You're trying to.

You need eyeballs, so you need news.

Every week there's not much news.

Obama's expected to win, but he might not. Every week Obama's in the lead, but he might lose.

That's what we knew. So it's hard. And one way of creating news is.

To focus on these noise fluctuations.

So if he shifted to saying things.

Like three and four chants, I think that's a good thing. He might have lost a few clicks that way. But one thing I've always admired for Many years about Nate is his integrity. And I don't think he would want people to get fooled by noise. So it's a very good thing that he's done that.

So moving to polling. Polling is generally thought of with respect to election forecasting. I'm wondering what polling is more generally and what type of things it can tell us.

Survey sampling is whenever you want to.

Learn about the whole from a part, right?

So a blood test is like a survey sample.

They take a sample of your blood and it's supposed to be representative of your blood. And if I interview people on the street and ask them how they're going to vote, then that's supposed to be representative of the general population. Well, it might not be.

They do random digit dialing.

That's kind of representative of the population, except not everybody answers the phone. Most people don't answer the phone, actually, so it's not at all representative of the population.

I was talking in my class, I was saying how I think it's mildly unethical to do an opinion poll and.

Not pay people because you do a.

Survey, you're making money from your survey. And a lot of pollsters do, like online panels, pay people, but a lot of telephone polls, they just call people up. I say, you're kind of abusing people's.

Goodwill by doing that.

And then someone said, but what about the kind of people who only participate in a survey because you pay? Are they non representative? And I said, what do you think about the kind of people who will.

Participate in a survey for free?

They're kind of weird people, huh? Most people don't. Most people hang up on pollsters. So survey respondents are not representative. We do a lot of work to.

Adjust the sample to the population. We need to because response rates are so low.

But anyway, it's not just election polling. It could be public opinion, as I said, it could be blood testing. It could be businesses use.

They audit their own records.

And if they want to do an.

Audit, they'll take a random sample of records and then audit the random sample and use that to draw conclusions about the whole business, et cetera.

So before we move into polling in a bit more detail, I'm just wondering, can you tell us why polling is even important?

Well, George Gallup, who was the founder of his poll, wrote a lot about this. He argued that polling is good for democracy. So there's two ways of putting it. Bill James, the great baseball analyst Bill.

James, once said something along the lines.

Of the alternative to good statistics is not no statistics. It's bad statistics. So what he was talking about was some like he was arguing that some.

Baseball player was overrated, blah blah, blah, blah blah.

And then he quoted some sports writers saying, oh, this Bill James number cruncher doesn't know anything. This batter was amazing. He hit.300 all these times and he bubbled. He got all these. And Bill James pointed out that. Let's look at what the sports writer said. What did he say? What was his evidence that this guy was such a great athlete? It was a bunch of statistics. He was just using statistics. So he was using statistics naively, but he was still the guy wasn't like being Mr. Qualitative. He started talking about how the baseball player was hitting.300. Similarly, suppose you are a legislator, for example, and you want to know about public opinion. Well, I think first public opinion is relevant. We don't always like when politicians follow public opinion too much, but I think.

We'D like them to be aware of public opinion. So if they don't have polls, what are they going to do? They might very well do informal polls.

Are canvassing and that over represents certain.

Kinds of people that's going to be unrepresentative of the people they find hard to reach.

Gallup's argument was pretty much that democracy.

Is ultimately based on public opinion and knowing public opinion between the elections is important. A lot of issues come up and it should allow politicians to do a better job, which the seems reasonable to me. Now then beyond all that, of course, surveys are used all the time in marketing so business people don't have to apologize for wanting to know what the customer wants. So it makes sense to do that. Marketing surveys are very interesting in part because you get into this question of connecting the observed measurements to what you really care about. Because is a marketing survey, how realistic is it? So if I call you up on the phone and say would you pay $30,000 for this sort of electric car? You could say yes or no. That doesn't mean that that's going to actually make it out of the showroom because the survey is not realistic. Political surveys are a little easier. Who do you plan to vote for? That's almost the same as being in the damn polling booth and voting. So the realism of political surveys is much closer to then the realism of certain marketing surveys.

And I don't know how long this has been the case, but we've definitely seen polling affect. There's a feedback loop into the political and voting and election process. I think it was so the primaries. Right. I think the Debates, your position on the stage and whether you're in the debate or not is actually dependent on your performance in polls, right?

Oh, yeah. And Donald Trump, like, he would always, when he would give speeches in the primaries, he would talk about how high.

His poll ratings were until they weren't.

And then he said, they're unscientific.

Well, yeah, but I mean, I'm not talking about his approval, but the percentage.

Of people who said they were going to vote for him. So he was polling very high, even when observers, outside observers, didn't seem to give him much of a chance.

So, yeah, there is feedback. I'll say one thing, that there's a useful feedback, at least for pollsters. Sometimes the question arises, why do people.

Tell the truth to a pollster?

And you'll sometimes get like, pundits will say, hey, let's all lie to the pollsters.

Let's screw them up. I don't like the pollsters, tell them.

The opposite of what you think. And yet people don't do that. And there's a couple of reasons for this. The first is that, as I said.

Polls are opt in. No one forces you to do a poll.

So if you really hate pollsters, it's.

Likely you won't bother to do it in the first place.

But the second thing is that I think people think of a poll as like, a way of voting. So if you ask if I survey you and do you approve of Donald Trump's job performance, you think this might get out in the news somewhere, you're motivated, if you approve, you're motivated to say yes. And if you don't, you're motivated to say no. There's a direct motivation to be sincere in your response. And again, that's not true of all surveys. If I ask you, do you take illegal drugs?

You might have various motivations not to answer that honestly.

And I couldn't answer that on air either.

Well, there's asymmetries.

Like you could if your answer was.

No, and you could answer it on air if you want. I'm not asking you, I'm just saying that it's complicated. So one thing that's not always well understood about political polling is that the.

Incentives align to actually encourage sensation in the survey response. That's very important.

Now, the other thing you mentioned that I just want to touch on briefly is this idea of polls measuring public opinion. And this is more playing devil's advocate, not necessarily trolling. But I'm just wondering, you know, public opinion generally is sort of, you know, views that are prevalent among the general public. And does public opinion even exist?

It's like Heisenberg's uncertainty principle.

So to measure opinion is to change it just that. You know how if you want to measure the position of a particle, you have to look at it, and looking at it means bouncing a light particle off of it, and that adds energy, and it changes its position and momentum. So similarly, if you want to know.

What someone thinks, you have to ask them. And then that changes now, not always.

You can observe their behavior.

So that's. There's other way. You don't have to. I have a colleague, Matt Salganik.

He's a sociologist at Princeton, and he wrote a book about social science data collection.

And he talked about, you can survey.

People, you can ask them, or you can observe them, and those are different. So you can actually observe someone without changing.

You can observe someone so unobtrusively, it.

Doesn'T change their behavior sometimes. So, like, Amazon can look at how you purchase. Now, arguably, once you know that Amazon's looking, then you might not purchase certain things or not search on things because you don't want them to know about it. But until that happens, you can observe them similarly, with the security camera outside your apartment. If you don't know it's there, then it's observing you pretty well. So in that sense, if you think of, like, we being measured, we're kind of in a cat and mouse game with those social scientists who are trying to measure us, that they're trying to measure us in ways that don't disturb us, and we might want to be aware of how we're being measured.

We'll jump right back into our interview with Andy Gelman after a short segment. Now it's time for a segment called Data Science Best Practices. I'm back here with Ben Skranker, an independent data science consultant. Hey there, Ben.

Hi, Hugo. It's great to be back on your show. I have a question for you. How do you know when you have the right answer?

Oof, I feel like that's a loaded question, Ben. Why do you ask?

Well, when I started out in physics, I too wanted to know if I had solved a problem correctly. I used to ask a cool grad student friend if I was right. He used to look at me like my question was ridiculous and would tell me, well, you just know when you're right. But I wanted something more concrete.

Well, if anybody has a more concrete.

Answer, I'm sure it would be a physicist.

You're right, Hugo. I was fortunate to work For Robert Rosner, the former director of Argonne National Laboratory, who's quite a polymath in addition to being an astrophysicist, he introduced me to what I was looking for. Verification, validation and uncertainty quantification, or VV&UQ. VV&UQ is an epistemological framework for thinking about correctness of models. It was developed in the nuclear industry where mistakes are expensive. Really expensive.

So how does it work?

Well, there are three parts. Verification, validation and uncertainty quantification. Like it says on the label, you pretty much move through them in order of maturity. Practitioners typically describe VV and UQ as the first view is did you solve the equations right? And the second V is did you solve the right equations?

That's slightly opaque to say the least.

I didn't find the VV and UQ insiders comment helpful either. For me, the first V verification means you need to make sure that your code correctly implements the model, regardless of whether the model is correct.

How do you do that?

Well, one of the best tools is to unit test your code. Many scientists and engineers think that it is impossible to test code which produces a number. But that simply is not true. You absolutely can and must test scientific code. Fortunately, you usually can if you are clever.

Can you provide an example?

Sure. I once wrote a unit test where I was able to pack a matrix with certain values so that I could compute by hand the answer. Our fancy algorithm would compute using Spark. So obviously if you're using Spark, you're dealing with something that's super huge. But by choosing clever values, I was able to compute an analytic solution and then verify it in the unit test. The person who checked my PR was blown away.

Great, so that's verification. What about validation?

The second V validation. Make sure your model has fidelity to reality. How you approach this depends on the problem. Often you can simulate data to make sure you can recover the original parameters from the data generating process. Or you can run an experiment and.

Then the mysterious UQ bit.

UQ is about questioning the assumptions behind the model. Make them explicit, check that they hold. What if they don't? Could you have a tsunami wipe out power for your nuclear power plant? Along what dimensions will your model break down? I know an economist who stress tests his work by taking it to a statistician who hates economists. Probably not the best UQ method.

And how do you suggest baking VV and UQ into a more general data science workflow?

Great question, Hugo. This is something you need to do and not enough people think about systematic workflows you should bake VV and UQ into your workflow. If you haven't thought much about workflow, Krisp DM is the best one I have seen in industry and we have already discussed it on Dataframe. If you combine VV and UQ and workflow, you will definitely be a 99th percentile data scientist.

Thanks Ben, for another great segment on.

Data science best practices.

You got it, Hugo.

Time to get straight back into our chat with Andy.

So now I want to get directly into polling and this is something that's known, I'm going to quote, quote you because you said it so well in an article in Slate magazine that we'll link to in the show notes. But you wrote the statistical theory of traditional polling is amazing. In the theory, a random sample of a thousand people is enough to estimate public opinion within a margin of error of plus or minus 3 percentage points. Could you just give us the rundown of what this exactly means?

This is the mathematics of drawing balls from an urn.

So if you have a large urn that's full of balls and 55% of the balls are green and 45% are yell, and you draw a ball at random a thousand times, then most likely you'll get between 52% and 58% white balls. So I said white, I mean green. So they're green and yellow.

And so it's 55% in the urn.

And you draw 1,000 each time you draw a ball and then you throw it back in the urn and shuffle it and draw another one.

Then the mathematics of probability tells you.

That the most likely thing you'll see is 55% green balls. But there it could be between anywhere between 52% and 58%. There's a 95% chance roughly that it's in that range. So we call that the margin of error. So if you can actually sample people like drawing them from an urn, then you can learn about public opinion pretty accurately.

But of course, this is theoretical. Right? And one of the parts of the theory is that it's a random representative sample. So I'm wondering what the practical problems and challenges associated with this theory.

Right, so in practice, you, you can't.

Draw people at random from the urn because, like, there's no list of people you can call phone numbers at random. Not everyone has a phone. Some people have two phones, some people never answer their phone, et cetera. Then also, if you draw a ball, you get to look at it in the EARN model. But when you're sampling people, you draw a ball. And what if they don't want to respond to your survey? Then you don't get to see it. So our surveys are systematically non representative of the population. So what we do is we adjust for known differences between sample and population. So our population is 52% female, but our survey is 60% female. We adjust for that. Our survey gets too many old people. It gets too many white people, gets too many people from some states and not others. Different surveys have different biases. Exit polls, I've been told, tend to oversample Democrats. Maybe it has to do with who's willing to talk to the exit poll interviewer. The kind of people who are willing to answer the phone are maybe different then.

The other thing is you need to.

Worry about getting honest responses or adjusting for inaccuracy in survey responses, which, as I said, is less of an issue for political polling, but comes up in other surveys.

Yeah, I'm really interested in this idea of calling people on the telephone because classically, you know, historically everybody. Well, a lot of people had landlines and you could do that. This isn't the case anymore. And my understanding is that there's legislation that means you can't automate phone calls to cell phones. Is that right?

I don't know exactly what the laws are about, what you can, what you can and can't do.

But, you know, it was all kind of just a window when Gallup started doing polls.

It was they knock on doors because a lot of people didn't have phones back then. So there was a certain period where a lot of people had phones. And in other countries, not everybody had phones either. But again, even if you could call everybody, so what? Because the respondents are not representative of the population.

So it's really the adjustment process that is really key as well.

Yeah, it's both. You want to get, you got to try to get a representative sample, even.

Though you're not going to get it.

Because you want your biases to be correctable. So if my bias is that I.

Have too many women, I can correct for that. Or too many old people.

If my bias is that I have too many conservatives, can I correct for that? Well, maybe because you can ask people.

Their party affiliation and then you can match it with data on people's party registration. It's more work. Right.

If I'm asking about health care and.

I have a bias, and my bias is that people have health problems are.

More likely to respond to the survey.

Can I adjust for that? Well, that might be tougher Right.

So it makes sense to try to.

Get that perfect sample, even though you're not going to get there to aim for it.

And are these correction and adjustment methods relatively sophisticated statistically?

Well, they're getting more sophisticated as our.

Data get worse and worse.

So the short story is that there's.

Three reasons they need to get more sophisticated. One is to adjust for inaccurate responses, but as I said, I'm not really going to focus on that. Second is differences between sample and the population.

You want to adjust for a lot of factors, not just sex and age.

And ethnicity, party identification, lots of things.

So when you want to adjust for.

More things, then simple adjustment methods, like simple weighting methods don't do the job. So you need to use like. So we use a method called multi level regression and post stratification. There are other approaches, but you need more sophistication to adjust for more variables.

And then the third thing is that we ask more from our surveys. So we might want to know, not just public opinion, not just do people.

Want to vote for the Democrats and.

Republicans, but how does that vote break.

Down among all of the 435 congressional districts? So even if I have big data, I won't necessarily have a large sample in each congressional district. So you want to do statistical analysis to get those more focused inferences. So that's one reason why we, my colleagues and I, have put a lot of effort into modeling the survey response is to be able to estimate subgroups of the population, like rich voters and poor voters, within different states.

Fantastic. And something that I know you've worked in is thinking kind of outside the box how to get people involved in surveys. That was an unintentional poor pun, but because you've actually used gaming technology and Xboxes in order to get survey responses, right?

Yes, my colleagues at Microsoft Research in.

New York did that.

So the Xbox gaming platform, Microsoft Research.

Has some social scientists, and my colleagues David Rothschild and Sharad Goyal, who worked there at the time, designed a survey.

So they convinced the Microsoft people to put something on the Xbox.

The last months of the 2012 presidential.

Election, where people could vote and say.

Who they wanted to vote for. So you could.

Every once in a while.

You get a reminder saying, would you like to participate in our poll? And then you'd give some demographics and say who you wanted to vote for. We had a huge sample size, hundreds of thousands of responses. Very unrepresentative.

An unusual survey because it overrepresented young men.

Most surveys overrepresent old women.

But after adjustment, we were first, we were able to estimate public opinion very well. In fact, we're able to estimate public.

Opinion more stably than public poll aggregators that had been out.

Now that's the good news.

The bad news is we didn't act. We took the collected the data in 2012. We didn't actually do the analysis until afterwards. So in theory it could have been done real time, but actually it was a research project and we published it later. So we didn't beat the polls while it was happening.

But not only that, we actually learned.

Something about political science and public opinion.

So we learned, as I said, our estimates were more stable and better than.

The polling aggregate estimates from the newspapers and online.

And it turned out that about 2/3.

Of the variation in the poll, the.

Fluctuations like Romney's doing well, Obama's doing well, these fluctuations, about 2/3 of those.

Fluctuations were actually attributable to differential non response.

So when Romney had some good news, Romney supported Republicans were more likely to answer the poll. Which makes sense, right? Like do you want to participate in a poll? Well, if my candidate is a laughingstock.

Maybe not if my candidate is like doing great stuff, yes.

So there was this feedback, a positive.

Feedback mechanism that negative feedback stabilizes. Positive feedback amplifies fluctuations. So positive feedback mechanism, which is if a candidate is doing well, more of their supporters respond to the poll, meaning they look like they're doing even better. And so you get these big fluctuations from week to week.

But then when you actually adjust for.

A partisanship, you find that the results were much more stable. And we found that in 2016 also.

Now you might say, well, maybe people's.

Partisanship is fluctuating too, but we have evidence that that's not really happening. So there's various loose ends in the project that we tied up when who wrote our paper. All of that came from this collaboration with these people at Microsoft.

I'm glad you mentioned 2016 because as you stated earlier, the popular vote, the pollsters did pretty, pretty well on within 1%, right? It was 52 instead of 51 was what the pollsters said. But of course, in the electoral college vote things were relatively different.

And I think something you've written about.

Is that that's potentially due to the fact that the in several key states, people who voted for Trump didn't necessarily respond in the polls. Is that right or do I misremember.

There'S so much non response? It's not so much. I mean, sure, I think the problem was more with state polls than national polls, that is even there were some people after the election there, some pollsters, Gary Langer and some of his colleagues wrote a paper where they analyzed their national polls by state and they actually found that the state level analyses of the national polls were not that far off. But the state polls, some of the state polls in Michigan and other states.

Didn'T do a good enough job of adjusting for non response.

So it seems there is a lot going on.

But part of it is that is that the non response adjustments weren't really as complete.

And it's an issue like it's survey.

Response rates keep going down. And so the raw survey data or even the weekly adjusted survey data are not always enough.

So in the same in the Slate article that I mentioned earlier, you also wrote instead of zeroing in on elections, we should think of polling and public opinion as more of a continuous process for understanding policy. I find this very attractive and I'm just wondering if you can elucidate this and tell me what you mean by this.

Well, this came about, I think it's particularly clear in the Obama administration that there were various issues like the stimulus plan, the health care plan, where public opinion seemed to be very important. There's a lot of both sides rallying.

Public opinion in order to sway certain swaying votes.

In Congress, it's less so right now. I mean, right now it's like, okay.

Republicans control the House, the Senate presidency and the Supreme Court.

And so it's sort of up to.

Them what to do. And public opinion doesn't seem to be directly influencing.

I mean they seem to be willing.

To do various unpopular things to make use of the majority they have.

But, but in most of the time.

Politics is at the legislative level, it's a bit more transactional. There are swing voters, certainly if one party controls the House and one party controls the Senate, then you get more power to various swing voters. And at that point public opinion can make a difference.

So it's not just about who you're.

Going to vote for, it's how people are going to vote once they're in office. So pollsters are going to be interested in public opinion throughout the process because it's not really just about who you plan to vote for, but also what are your views on various issues, whether it's foreign policy or health care or immigration or trade or whatever.

And how does party allegiance play a role in that?

Party allegiance is very important.

There's a lot of evidence that voters.

Will switch positions based on what their party says.

Even if you look at things like support for wars. There's been just big jumps based on the party in power. I mean, people just like, if you're a Democrat, then you support the same policy that you wouldn't support if a Republican were stewing or, or vice versa or how things are labeled. But it's like, as the economists say, it's exogenous. Pollsters. Pollsters are measuring opinion, but at the same time, politicians are trying to use that opinion. My colleague Bob Shapiro in the political science department, he and a colleague wrote a book called Politicians Don't Pander, which.

Was based on his study of various.

Political fights, not elections, but legislative fights.

And he argued that politicians think of.

Public opinion as a tool, that there's a naive view of politicians wanting to do what the voters, what public wants.

But it's actually politicians often are quite.

Confident and they feel that they can sway the voters and they think of public opinion as something that they can manipulate. And so both sides are doing it. And so there's a, there's.

And to the extent that individual Congress.

Members and senators are involved, you also need to know about local public opinion, not just national.

So, Andy, what does the future of polling look like to you?

I don't have a great sense of what the future will be like. If you look traditionally, you'd say, well, lower response rates.

Is the future paying people to participate online panels?

I think maybe in general we should.

Think of the people who respond to surveys as being more participants, the same.

Way as like medical statistics.

Instead of thinking that we're measuring people and estimating the effect of the drug and people are just like counters that are being moved around.

We should actually think of patients as.

Participating in studies and really being involved.

Not just because you want to get more compliance, but also because people have.

A lot of private knowledge that they can share as well as that if people should be more motivated to help out, if they have more connection. So to me, it would be the future would be something a bit more.

Collaborative in the other direction.

There's just going to be a lot of passive measurement, like things like Amazon measuring your clicks. That's like polling also. So that's from the opposite direction. So either if it's intrusive, I think people should be more involved, or it's just not intrusive at all.

So, Andrew, my final question for you is, what's one of your favorite data, sciencey and statistical techniques or methodologies?

You know, my favorite thing is something.

That I've never done and I read about.

So this was maybe 10 years ago, supposedly someone built a machine that you could stick in someone's office and then if they're typing 10 minutes later, it.

Could be a keylogger. And supposedly the way it works is There are about 100 keys on your keyboard. So it listens to the sounds and uses some classification algorithm to classify the sounds of the keys into 100 clusters. And then having done that, it then uses simple code breaking techniques to estimate like which is the space bar, which is the carriage return, which is the letter E, and so forth.

Of course, it doesn't have to be perfect.

You can use statistical tools and then it can figure out what you're typing.

So I've always wanted to build that. Now that kind of thing I don't.

Know how to build. I mean, like it also involves having a microphone and doing sound analysis. I just think it would be really cool. These things are pretty Bayesian. You're using a lot of prior information on, especially the second step, the code breaking step. Of course, Alan Turing used Bayesian methods to crack the Enigma code in World War II.

I shouldn't really.

That's my favorite example, even though I've never seen it. I just think it's the coolest.

It's not something that, that I could really do though. So I guess in that if you.

Want to talk about stuff that I.

Could do, then my favorite technique would.

Be multilevel regression and post stratification because that's what we use to estimate state level public opinion. It's how we did red state, blue state, and estimated how opinions vary by income in different parts of the country. It allows us to do our best to adjust for differences between sample and population. We can do it in Stan. So I'll push that. Great.

And so multi level regression and post stratification will include some links in the show notes. It's also known as MrP or Mr. P. Correct.

Exactly.

And then recently I started calling it regularized prediction and post stratification because strictly speaking it's modular. So multilevel regression is the first part.

Is you fit a model to do the adjustment.

The second part is having done that.

You make inferences for the population, which is called post traffication. So multi level regression is one way of doing the model, but generally you could use the term regularized prediction, which includes all sorts of other methods that are out there.

It's been such a pleasure having you on the show.

Likewise.

Thanks for joining our conversation with Andy. We saw that the three biggest challenges facing statistical inference and data science are the same that they have been for some time, and that these challenges are also the main issues faced by polling and election forecasting. The first is generalizing from the sample you collect to the population of interest. The second is generalising from the control to the treatment group, which we only have in rare cases. The third is generalizing from your observed measurements to the underlying constructs of actual interest. We also saw the importance of thinking of polling as a more continuous process of gauging public opinion and understanding policy as opposed to focusing merely on elections. On top of this, we saw the need to get creative with polling methods in an age in which responses are dwindling. One example of which is using xboxes for polling and then performing sophisticated correction methods such as Mr. P to correct for your non representative samples. Also, make sure to check out our next episode A conversation with Brian Granger about data science, interactive computing, open source software and Project Jupyter. With over 2.5 million public Jupyter Notebooks on GitHub alone, Project Jupyter is a force to be reckoned with. What is interactive computing and why is it important for data science work? What are all the moving parts of the Jupyter ecosystem, from notebooks to Jupyter Lab to Jupyter Hub and Binder and beyond? And why are they so relevant as more and more institutions adopt open source software for interactive computing and data science? Tune in next week to find out from Brian, co founder and co lead of Project Jupiter, Physicist and co creator of the Altair package for statistical Visualization in Python. I'm your host Hugo Bown Anderson. You can follow me on Twitter at hugobown and Data Camp At Data Camp you can find all our episodes and show notes@datacamp.com community podcast.