Philip E. Tetlock on Forecasting and Foraging as a Fox | Conversations with Tyler

Today I am speaking to Philip Tetlock, who quite simply is one of the greatest social scientists in the world. Up until now, we've been doing conversations with Tyler face-to-face, but for obvious reasons, Philip is in Philadelphia. He teaches at University of Pennsylvania, and I'm here in Arlington, Virginia. Let's just jump right into it. First question, Philip. With our forecasters, do we want accuracy or do we want them to be a kind of portfolio to make us more aware of extreme events and possibilities?

I think we want a lot of things from our forecasters, and accuracy is often not the first thing. I think that we want— we look to forecasters for ideological reassurement— reassurance. We look to forecasters for entertainment.— and we look to forecasters for minimizing regret functions of various sorts. So that we would really regret not having anticipated X, Y, or Z, so we want to pump up the probabilities of those things.

But if we take, say, the coronavirus, if we had had a few more extreme nuts who were maybe wrong most of the time, but insisting that we needed to fear the next pandemic, wouldn't we have been better off with that kind of portfolio?— and thus we don't actually want more accuracy from our forecasters.

Well, in some sense, we already did have that portfolio. There were— it was a mainstream position among epidemiologists for the last 20 years or so that you had a recipe for disaster. David Epstein, the guy who recently wrote Range, a very interesting guy— you may have had him on your show, I don't know— but he recently quoted himself from a 2007 newsletter that he wrote. And he said something like, the presence of a large reservoir of SARS-like viruses among horseshoe bats combined with a culture of eating exotic meats is a time bomb. But what's the problem? And that was in microbiology books. That was in the first decade of the 21st century. It was common knowledge. And indeed, even before SARS-1, even before the first SARS outbreak— and some epidemiologists prefer to call COVID-19 SARS-2. But even before SARS-1, epidemiologists were acutely aware of this. So, it's not as though we didn't have it in our portfolio. We did.

So, those forecasters maybe weren't entertaining enough? Isn't then the margin we want to work on to make our better forecasters more entertaining and not more accurate? Yes? No?

Well, the signal-to-noise ratio isn't going to be great. I can assure you of that because there are plenty of people who are naturally more entertaining than epidemiologists.

But maybe the whole portfolio needs to be more vivid rather than trying to fine-tune the accuracy of particular parts of it, right?

Well, you have lots of people competing in the marketplace of ideas for attention, and that's a hard competition for scientists to beat.

What do you think of the argument that science only exists at all because most scientists are overconfident, that if they were rational Bayesians, they would just latch on to the opinions of the smartest and best-trained people before them, that there's only progress precisely because people are making forecasting mistakes.

Right, so you hear versions of that argument, I suppose, on Wall Street as well.

Sure.

That's— I think there's a good deal of truth to it. I certainly have been guilty of overconfidence at many junctures in my career, thinking I'm going to be able to take on things that looked impossible and often turned out to be impossible. Most projects that most scientists embark on, I think, don't succeed. It does take a certain amount of quasi-irrational persistence.

As with Columbus, right? Or the founding of the United States. Arguably was irrational to break away from the British Empire, right? Which was doing pretty well back then.

It was, it was, it was a, it seemed like a risk-seeking move.

But say I set up an alternate research program and I sought to take forecasters and A, make them more entertaining, and B, maybe I'd give them uppers so they were more overconfident. I mean, would that do the world good?

Well, you could certainly have induced the epidemiologists who are worried about the horseshoe bats in central China or other possible sources of zoonotic viruses. You could certainly have induced them to pump up their probabilities.— and you would, of course, start to run into a problem of crying wolf. If they'd been saying there's a 30%, 40%, 50% chance of a viral leap into human beings each year, and it didn't happen, didn't happen, didn't happen, people would— you get a crying wolf effect, right? Right.

How do you think about financial markets in relation to your work on super predictors? Are financial markets, in essence, super predictors to begin with? Or can super predictors on average beat financial markets?

Oh, boy. Well, you know, we play with prediction markets in the work with the intelligence community. Going all the way back to Admiral Poindexter and the original DARPA effort to launch prediction markets inside the intelligence community, I think your colleague Robin Hanson was involved with some of that work around 2000. I would say around 2003, 2004, in that range. We've also been working with the prediction markets in parallel with forecasting tournaments, and there are pros and cons to each method of eliciting judgments. But the IC, the intelligence community, doesn't let us turn those markets— make those markets deep and liquid the way they are on Wall Street. People are essentially competing for reputational points the way they are in forecasting tournaments. The monetary prizes are either small or nonexistent. So most economists, I think, would not consider that to be a very robust test of the efficacy of prediction markets, and then when prediction markets fall short— and they do typically not perform quite as well as forecasting tournaments— when they fall short, it's hardly a decisive rebuke of the market mechanism for eliciting forecasts. There are, there are lots of very powerful institutional actors like Goldman Sachs and so forth that are continually trying to do exactly what you describe. It's an ongoing process. I don't need to— you're an economist, I don't need to tell you that.

Sure. I mean, there are implicit prediction markets in, say, coronavirus, say, prices of airline stocks, right? And those are very liquid. They've been very thickly traded lately.

They have indeed.

If you took your 10 best superforecasters and brought them into the hedge fund people at Goldman Sachs, and you all sat down together, Who would be teaching whom?

It's an interesting experiment. And the person— OK, my project manager from the first set of forecasting tournaments, Terry Murray, founded a company, Good Judgment Incorporated, which does things like that. So that's a proprietary venture. And you probably want to talk to Terry about how successful or not successful they've been in doing that. I think they've had some success. I think it's extremely hard to do that.— it's non-trivial. I think there's a good deal of similarity in the cognitive ability, cognitive style profiles of superforecasters and the kinds of people you see on the staffs of Goldman Sachs, and it would be a tight race.

What about the sports betting market? Do you think there are inefficiencies in that because they don't have enough superforecasters?

I'm not an expert on sports betting. Better to talk to Nate Silver about that.—

But let me put the question more generally. There are many markets out there which predict something. Sports betting markets are simply the most obviously, most explicit about prediction. So, if there was something the world didn't know about prediction already, those markets should be inefficient. Yes or no?

Why would you think the answer would be yes?

Well, let's say that people favored the home team too much. So, too many people might bet on the New York teams, the Los Angeles teams, and then the odds would be skewed. So, if there's a bias in people without superforecasting techniques, we would expect sports odds to somehow be off, or at least they would have been off before your work was published.

Well, arbitraging predated superforecasting.

Well, but isn't arbitrage itself superforecasting, right? People are arbitraging—

Yes, in a form, yes.

—on the basis of some set of information.

I think that's fair, yeah.

So, all these markets out there, do you think they are, without you already, the best available superforecasters?

The term best is a term I'm just not comfortable with. I rather doubt that they're at the optimal forecasting frontier at the moment, but they're often probably fairly close. And is there room to incentivize people to outpredict the market? Well, that's one of those paradoxes that economists have written about, right?

When people predict out to many decimal places, do you think that's absurd or do you think it's useful?

What's the optimal level of granularity for different categories of forecasts? I think for the kinds of things we were looking at in the IARPA original forecasting tournaments with geopolitical events, like how long the Syrian civil war would last, or what would Russia do in eastern Ukraine, or things of that sort, yes, it would be absurd to go to 3 or 4 decimal points. Originally, the National Intelligence Council, which synthesizes a lot of intelligence analysis, originally they only distinguished 5 degrees of uncertainty and they didn't, they didn't put numbers on it. More recently, they've moved to 7 degrees of uncertainty and they do put numerical ranges on. So, you know, somewhat likely represents a certain probability range. Now, in our work, we've explored how granular the best forecasters are, doing various rounding experiments where we round their forecasts off to the nearest tenth, that kind of thing, and see whether or not— if it's pseudo-precision, if when they adjust, when they move from 0.6 to 0.65, for example, if on average that doesn't improve their accuracy, we would conclude that they can't achieve that level of granularity. Our best statistical estimates are that our forecasters for the types of questions the intelligence community often poses can distinguish between 10 and 15 degrees of uncertainty, which is considerably more than the 7 they think they can now, a lot more than the 5 they thought they used to be able to distinguish. But how useful is it to be able to distinguish varying degrees of uncertainty is going to hinge on the kind of game you're playing. If it's poker, you might— —well, on someone who's very adept at distinguishing things to, say, 3 decimal points.

If you could take just a bit of time away from your research and play in your own tournaments, are you as good as your own best superforecasters? I don't think so.

I don't think I have the patience or the temperament for doing it. I did give it a try in the second year of the first set of forecasting tournaments back in 2012.— and I monitored the aggregates. We had an aggregation algorithm that was performing very well at the time, and it was outperforming 99.8% of the forecasters from whom the composite was derived. So, if I simply had predicted what the composite said at each point in time in that tournament, I would have been a super, super forecaster. I would have been better than 99.8% of the super forecasters. Even though I knew that it would be— it was unlikely that I could outperform the composite, I did research some questions where I thought the composite was excessively aggressive. And I tried to second-guess it. And the net result of my efforts, instead of finishing in the top 0.02% or whatever, I was— I think I finished in the middle of the superforecaster pack. So, that doesn't mean I'm a superforecaster. It just means that when I tried to make forecasts better than the composite, I degraded the accuracy significantly.

But what do you think is the kind of patience you're lacking? Because if I look at your career, you've been working on these databases on this topic for what, over 30 years? That's incredible patience, right? More patience than most of your superforecasters have shown. So, is there some disaggregated notion of patience where they have it and you don't?

Yeah, they have a skill set. I mean, and in the most recent tournaments we've been working on with them, this becomes even more evident. That their willingness to delve into the details of really pretty obscure problems for very minimal compensation is quite extraordinary. They are intrinsically cognitively motivated in a way that is quite remarkable. How am I different from that? I guess I have a little bit of attention deficit disorder, and my attention tends to roam. So I've not just worked on forecasting tournaments. I mean, I've been fairly persistent and pursuing this topic since the mid-1980s, you know, back when even before Gorbachev became general party secretary, I was doing a little bit of this, but I've been doing a lot of other things as well on the side. And so my attention tends to roam. I'm interested in taboo trade-offs. I'm interested in accountability. There are various things I've studied that don't quite fall in this rubric.

Doesn't that make you more of a fox though? You know something about many different areas. I could ask you about antebellum American discourse before the Civil War and you would know who had the smart arguments and who didn't, right?

Well, I would know who has arguments to take the more integratively complex forms on the one hand, on the other hand, and then synthesis. Whether you want to consider those arguments smarter or not is another matter. But yes, I suppose that's fair. I mean, I've always resonated a little more to the foxes than to the hedgehogs, but I think when you look at the great achievements in science, they often come from hedgehogs.

If you look today at the ongoing debates about coronavirus and what will happen, and I mean now the debates amongst the smart people, the people you respect, what is the mistake you see them making? The biggest mistake?

Oh, you want me to be an amateur epidemiologist here, right? No, mistake in reasoning.

You don't have to give your numerical estimate. Procedurally, what are they not getting right?

Well, is it a mistake if you're a public health expert who feels that one mistake is much worse than the other? It's much better to overestimate the threat of the virus than to underestimate it because you have to influence public opinion and public behavior. There you're not forecasting, you're engaged in manipulation, social influence. So you— I mean, this comes back to your original question about what do we want from our forecasters. And accuracy is only one of the things we want from them. We look to forecasters for a lot of things, whether to inspire confidence, to inspire fear, and so forth.

If we're trying to estimate how much people cut back on their risk-taking behavior because they're afraid of the virus, are the group of people best suited to do that epidemiologists, some other social scientists, or your superforecasters? Maybe even economists, right? We study elasticities. Why should it be the epidemiologists? I think you'd want an interdisciplinary team.

I think there's a— diversity is one of these words that's been reduced to a cliché, but I think we have found in our work that cognitive diversity helps.— and it helps in certain quite well-defined ways. If you want to create a composite that outpredicts the vast majority of the superforecasters, a good way to do it is not only to take the most recent forecast of the best forecasters in a domain, but it's also to extremize that forecast to the degree that people who normally disagree agree with each other. And when you have that— when you have convergence among diverse observers, that's a signal that the weighted average composite is probably too conservative and you should extremize.

Does the team, the diverse team, have a CEO, someone in charge?

Not in this case, no. That's done purely statistically.

But would you put someone in charge? Now maybe the person in charge would implement that statistical algorithm, right? But who's the person you would put in charge of the team? An epidemiologist, yourself, your best superforecaster, Bill Gates?

That's a matter of managerial skill. And I would say, going back to my old dissertation advisors at Yale, 40-plus years ago, Irv Janis on groupthink. I would pick a leader who knows how to shut up and not reveal opinions at the beginning of the meeting and knows how to listen.

And which group of people do you think that best describes?

Um, I think a lot of good executives have the intuition that you get more out of a team of forecasters or problem solvers if you initial— if you elicit independent judgments initially that are uncontaminated by conformity pressure and then you create an environment in which ideas can be freely critiqued before lifting the veil of anonymity and letting people see who's taking which positions.

Do you think having machine learning and artificial intelligence has made us much better at forecasting things right now?

I mean, social things for sure.

Sure, at the micro level, but social events, whether there'll be a recession, how many people will die from the coronavirus, Will we settle Mars?

Well, IRPA just ran a forecasting tournament called Hybrid Forecasting Competition in which they pitted algorithmic approaches and human approaches and hybrid approaches against each other. And, you know, I should let IRPA speak for itself about how well its programs work or don't work. I don't think there's a lot of evidence to support the claim that machine intelligence is well equipped to take on the sorts of problems that the intelligence community wanted to have answered when it runs the forecasting tournaments it's been running with our research team. Things like the Syrian Civil War, Russia-Ukraine, settlement on Mars. Though, these are events for which base rates are elusive. It's not like you're screening credit card applicants for Visa. Machine intelligence is going to dominate human intelligence totally. Machine intelligence dominates humans in Go and chess. It may now dominate humans in poker. I don't know that we're at the state of the art is quite there yet, but, you know, it is in their StarCraft or, you know, whatever the next thing that Dennis Asabis is going to conquer. So, but no, I don't see evidence that those approaches work in the domains that we study with the intelligence community.

So do you think the hybrid man-machine approaches are overrated?

No, I don't think it's, I think it's a matter of, it's very domain specific. It sounds like a great idea. Who could be against it? But the devil lurks in the details and it doesn't deliver as automatically as you might hope. It's trench warfare here.

Do you think the world as a whole is becoming easier to predict or harder to predict? And again, I mean social events.

I'm not sure there's a definite trend one way or the other. You hear a lot of talk, a lot of claims that there are, but if you look back on the 20th century, there certainly were lots of major pockets of unpredictability. It's not clear.

What if I say, again, current events aside, but it seems easier to predict. There hasn't been a world war since 1945. There's been steady economic growth in most parts of the world. More peace. Isn't that easier to predict? You just predict 2 to 4% global economic growth and you pick up a fair amount of what's happened since 1950.

Indeed. So, well, simple extrapolation algorithms historically are hard to beat. And if you— we just were running a COVID-19 mini forecasting tournament right now, and they're proving to be hard to beat. The skill, of course, is when to alter the trend, whether to accelerate it or to decelerate it or to change direction.

And do you have a personal intuition on that?

Mm, uh, I, I think humans have been repeatedly humbled in competitions against, uh, simple statistical algorithms, uh, going back to Paul Neal's famous little book on clinical versus actuarial approaches to predicting, um, in medicine and psychiatry. Um, so I would say be humble.

Now, there's some of your early research that, if I read it properly, suggests that making people accountable leads to more evasion and self-deception on their part. Are you worried that your work with pundits, by trying to make them more accountable, will lead to more evasion and self-deception from them? Or how do you square early Tetlock and mid-period to late TATLOC?

It's actually not too difficult in that particular case. It really depends on the type of accountability. Tournaments create a very stark monistic type of accountability in which one thing and only one thing matters, and that is accuracy. And you get no points for playing to your— for being an ideological cheerleader and pumping up the probabilities of things that your team wants to be true or downplaying the probability —these are the things your team doesn't want to be true—you take a reputational hit. So the incentives are very unusually tightly aligned to favor accuracy. That's extremely unusual in the social world. Most forms of accountability occur in organizational settings in which there are lots of distortions at work. And the rational political response for a decision-maker located in most accountability matrices in organizations is to engage in some mixture of strategic attitude shifting toward the views of important others, or as you put it, the evasion, procrastination, and so forth.

But given that when a pundit is on, say, the evening news, there's not a little box at the bottom that gives the Tetlock score of that pundit, correct? So most people don't know the actual record. So given that out there somewhere is a measure of how good or bad the pundit is, uh, are you worried that in a sense you will make those pundits run further away from objective standards precisely because they do poorly by them.

I don't know if we can make them run much further away from objective standards than they already are, but that's a very interesting point about forecasting tournaments. And I look at the kinds of people who are attracted to participate in them. At the very outset, I invited lots of big shots to participate in forecasting tournaments, and they turned me down. They've repeatedly turned me down.— had a very interesting correspondence with William Safire in the 1980s about forecasting tournaments. We could talk a little about later. But the upshot of this is that young people who are upwardly mobile see forecasting tournaments as an opportunity to rise. Old people like me, aging baby boomer types who occupy relatively high status inside organizations, see forecasting tournaments as a way to lose. If I'm a senior analyst inside an intelligence agency and I'm— say I'm on the National Intelligence Council, I'm an expert on China and the go-to guy for the president on China. And some upstart R&D operation called IRFA says, hey, we're going to run through these forecasting tournaments in which we assess how well the analytic community can put probabilities on what Xi Jinping is going to do next. And I'll be on a level playing field competing against 25-year-olds. I'm a 65-year-old. How am I likely to react to this proposal, to this new method of doing business? It doesn't take a lot of empathy or bureaucratic imagination to suppose I'm going to try to nix this thing.

Which nation's government in the world do you think listens to you the most?

You know, I— You may not know, right? I might actually. What can I say? You know, I suppose the most prominent political fan I have at the moment is probably one of the senior advisers to Boris Johnson, Dominic Cummings, who recently caused a bit of a stir in the UK by appointing a superforecaster who had written some blogs that people interpreted as misogynist or racist or fascist or eugenicist or I don't know, some mixture of all those things. But I think his name was Andrew Sobisky. He was a young man, 24 or 25, the kind of person young people are attracted, as I said before, to forecasting firms. It's a fast track toward upward mobility because you can have all the high-status people making vague verbiage forecasts, people like me.

But do you think Cummings actually is influenced by you? Because as I understand what he's doing, correctly or not, he thinks he knows a bunch of things that other people do not, and that seems somewhat non-Ted Lockean, right? So maybe you're part of his portfolio of ideological armor. Maybe he's actually very non-Tetlockian and it's the people in Singapore who are your true fans.

Well, there are some people in Singapore too, and that's an interesting— it's an interesting place. You should mention that. Interesting you bring that one up. But, well, Mr. Cummings and Mr. Gove both separately brought up my work at various points and during the Brexit debate. And Michael Gove, at least in the UK, famously said that Britain had had enough of experts. I don't know if you remember that— Of course. —particular quote. And he was thinking of— well, he invoked as support, at least, for that position, Expert Political Judgment, in which portions of that book compare subject matter experts to minimalist statistical baselines, like extrapolate. Can you predict simple extrapolation algorithms? And the answer was often no. Gove was raising the point that, you know, where do these guys get off making these confident predictions about the consequences of Brexit when the best empirical evidence would suggest they're probably not materially more accurate than simple extrapolation algorithms? So it was, you know, that was brought up. It was brought up for a political reason. He had a political point to make. And I think this— and Dominic Cummings had a pleasant political point to make as well. He fears that, I think, parts of the civil service are hostile toward Brexit and want to undermine the Boris Johnson administration objectives. Now, I think this comes to something deeper now. It's not just about the UK. It's about the intellectual fissure that exists between social science and conservatives, that most social scientists are liberal, and conservatives are wary of advice from social scientists. I think we may have partly paid some price for that. In this epidemiologic— in this, to put it kindly, the slowness of the Trump administration response to COVID-19.

Yes. Now, you brought up Britain. If we look back at speeches in the British House of Commons, who is giving the most cognitively complex speeches?

I can't— you know, Bob Putnam collected those data that I reported that study in 1984. But Bob Putnam collected the original data, and he reported it in a book. Beliefs of Politicians, published in the 1970s. And I think those data were based on interviews with members of the British House of Commons. They were not speeches. They were interviews, confidential interviews that Bob Putnam got access to and put a lot of work into obtaining. And he shared them with me. And I used them as grist for a small research program I was running in the 1980s on cognitive style and political ideology. Trying to tease apart the rigidity of the right versus the ideologue hypothesis.

And who sounds the smartest from that period?

In that period, it was a mixture of moderate labor, moderate conservative. It was the centrists that did better. It was slightly left shifted.

Do you think we can draw any inferences from that about politics? Should we have more faith in the people who sound smarter or not at all?

Well, I think That particular measure in integrative complexity is— I think it does have some correlation with forecasting accuracy, but I think you're picking up something more than just forecasting accuracy. You're picking up what I call value pluralism. You're picking up a tendency to endorse values that are often in conflict with each other. So, the more frequently that you, as a political thinker, confront cognitive dissonance between your values, your value orientations, the more pressure you are to engage in integratively complex synthetic thinking. Uh, when, when, when your values are more lopsided, it's easier to engage in what we call simpler modes of cognitive dissonance reduction, like denial or bolstering and spreading, spreading of the alternative. So we downplay one value, push up the other value and make your life stress-free. Um, but some value, some ideological positions at some points in history, um, require more tolerance for dissonance, and some people are more inclined to fill those roles.

And you think those politicians are also likely to be better forecasters?

You know, we're not talking about huge effect sizes here. Sure. Integrative complexity, I think that fluid intelligence is probably a more powerful predictor, but integrative complex— a combination of fluid intelligence and integrative complexity, I think does boost forecasting accuracy, yes.

—And if you were running the CIA, those people who are pluralistic in the manner you just outlined, would you promote them more rapidly? —Yeah.

Now, of course, the CIA, bear in mind, is by statute supposed to be value-neutral. It's not— they're supposed to be feeding impartial apolitical advice, justifying— —No one believes that, right? —Well, that is supposed to be the division of labor here. Now, is it— and forecasting tournaments are— I think one reason they may be interested in forecasting tournaments is because forecasting tournaments incentivize people to do one thing and only one thing, and that's accuracy. You don't get points for skewing your judgments toward your favorite cause. You take a hit on the long term by doing that.

If you were in charge of the CIA and had a free hand, how would you reform it?

Oh boy. Well, there's a long history to efforts to reform the CIA going back to 19— well, it was founded in 1947. There have been various efforts since then. People have been unhappy with the CIA for many reasons over time, Vietnam being a big one, but there are lots of other reasons that people have expressed unhappiness. Both liberals and conservatives have at various points been unhappy with the performance of the intelligence community. In 2001, it came to a kind of a crisis point, I think, and then there was a commission to reform intelligence analysis and after the WMD fiasco in Iraq that the pressure grew even more. One reason why we're even talking right now today is that the intelligence community was forced essentially by the recommendations of the, of the reform commission to take keeping score more seriously, to take training for accuracy and monitoring of accuracy more seriously. And I think that they create— that's when they created the Intelligence Advanced Research Projects Activity, which is the R&D branch housed within the Office of the Director of National Intelligence, and its job is to support innovative research that will improve the quality of intelligence analysis, where, you know, that can be defined in various ways, but accuracy is certainly one very important component of that, but it's not just— it's not accuracy with a liberal skew or a conservative skew. It's supposed to be just plain, just the facts, ma'am. Interesting.

But what questions they choose to predict, right? That reflects values, obviously. I'm sorry, which questions they choose to predict, that reflects values. Whenever a bureaucracy tells me they're value-free, I start getting more suspicious very quickly, right? I never believe them. It might be okay for them not to be value-free, but the cynical response is the correct one here.

Well, indeed, you know, the old expression, there's no view from nowhere.— there is no such thing as pure value neutrality. That doesn't mean it's not something worth aspiring to, but you're right. The values— even if you had a perfectly objective forecasting tournament system, if you had people generating the questions promoting a political agenda, you could skew the results. I think that's one of your points, right?

Now, in the middle of all these discourses, we have a segment called Overrated versus Underrated. And I'll toss out a few names, ideas, and you tell me if you think they're overrated or underrated. How's that? Okay. Philadelphia, the city of Philadelphia, overrated or underrated?

I got endless grief when my wife and I decided to leave Berkeley and move to Philadelphia. People thought that we were borderline insane. But we left for very personal reasons, and without going into what what those were. I would say Philadelphia has been a moderately pleasant surprise. It's a city that has many, many, many problems, but it's not as bad as the people in Northern California thought it was. Tolstoy. I haven't— I've read a little bit of Tolstoy and I've seen a number of films. I know some of the shorthand versions and I know that Isaiah Berlin had a hell of a time class telling tall stories as a hedgehog or a fox. But I think I'll pass on that one.

John Cleese. He's commented on you. You're allowed to comment on him.

Right. Well, he gave me— I had a wonderful time as a kid watching Monty Python and also Fawlty Towers. So I really enjoyed his comedy. I haven't followed him since then. But when I was younger, I thought he was absolutely hilarious and brilliant, and I appreciate the flattering things he said about my work.

The television show The Sopranos.

I fell for it, and I love James Gandolfini. I fell in love with his performances, and then when he, you know, one of the last things he did is he played Leon Panetta in Zero Dark Thirty,— and there he is eliciting forecasts from people in the CIA about whether Osama bin Laden is in that compound in Abbottabad. Is he there or isn't he effing there? Great, wonderful guy. I never knew him, but I think very highly of his work.

The threat of terrorism, do we overrate it or underrate it in the United States?

That's a very difficult question because of the tail risk aspect to it. If you looked at the number of people who died from terrorism versus other causes, it would seem that the amount of money we spend on suppressing terrorism would be disproportionate. But the tail risk complicates that a lot.

What is your favorite movie?

I don't have a hierarchy like that. I'm sorry.

Nothing comes to mind? What's the movie you've seen the greatest number of times? That you can count.

I don't know if I watch movies all that often over and over. I did see myself coming back just very recently, last week in the quarantine period, to Westworld. It's not a movie, it's a series, of course, but I think that is quite— I think the first two seasons are quite brilliant.

On historical counterfactuals, By what year do you think the ascent of the West was more or less inevitable?

Well, I have inside information here. We did a survey of some very prominent historians. Yes. We reported it in that book on Making the West. We reported part of it anyway, also in an article on American Political Science Review. I think that if you looked at just the unweighted average of judgments for the median, I think was probably around 1730, 1740.

And you think after that it was not very contingent?

That would have been— you see, I'm not a historian of the West. I mean, what do I really know about that? I'm simply reporting the news here.

And is there any great hinge of contingency that you think about looking backwards? Like, oh my goodness, if there hadn't been a Reformation, or if there hadn't been a Council of Trent, or or what?

Well, I'm a fan of Steve Pinker. I think the Enlightenment was a big deal because it got people thinking more rationally.

Yeah. And more in terms of science.

And that had huge, huge spillover effects. Now, could there have been an Enlightenment in China? Or could the Chinese have created certain types of technologies without science? What if they hadn't scuttled their navy around 1400? There are all those sorts of counterfactuals.

Um, and how necessary, how necessary or contingent, contingent do you think it is that we keep on thinking in enlightenment-like terms? Is it once you're locked into it, it keeps on going? Or is it like good government that you have to renew it every generation or two?

Well, I see the work I'm doing is very much in the spirit of the Enlightenment. Uh, public transparent standards of evidence for judging, uh, subject matter expertise. I think one of the great challenges of our time is striking the right balance between democracy and technocracy. And I think the fissures that have emerged, it's not just conservatives are very skeptical of social science and it's apparently some parts of biological science too, which is I think very unfortunate. But you see this kind of some reflexive skepticism towards science on the left as well. So, I mean, how do you manage the relation between small d democrats and technocrats in a society in which expert guidance is increasingly crucial.

What are you learning from playing the game Civilization V, or at least watching others do so?

That's a good example, actually, of why I'm not a superforecaster. I don't play Civilization V, but Civilization V was one of the simulations that IARPA chose to feature in its counterfactual forecasting tournaments under the rubric of Focus. If any of your listeners are interested in signing up to be forecasters for Focus, we still have one more round, round 5, and we will be recruiting people. Again, it reflects my temperament. I don't have the patience for a game like Civ V. That requires a lot— Forecasting is inevitably a mixture of fluid and crystallized intelligence. You have to invest a lot of energy into mastering a game like Civilization V. And I suppose as people get older, they may become less likely to make those kinds of cognitive investments. I mean, it becomes more and more essential as I get older, I think, to focus on the things where I have a real comparative advantage.

The best chess players are all young, right? This we know.

There's clear data. Well, yeah. I mean, it's interesting the domains in which child prodigies emerge. Music and chess and math.

If we take super counterfactualists and super forecasters, do those two groups basically overlap or how do they differ?

I think they have to be rather intimately connected, although disentangling this one is going to be really, really hard and it's one of the things I do want to dedicate a few years of my life to doing. Now obviously— when you say someone's a good counterfactualizer, people shrug their shoulders and they say, well, how are you possibly going to know whether you would have gotten— undo the assassination of the Archduke in 1914, you undo World War I. Undo Hitler, you undo World War II. You make Kennedy grouchier during the Cuban Missile Crisis, you trigger World War III. You've got these sorts of arguments that are essentially unresolvable.. You can't rerun history. You can rerun Civilization V, but you can't rerun history. So, that makes counterfactuals a place where ideologues can retreat. They can make up the data, make up whatever facts they want to justify pretty much whatever— no matter how bad the war in Iraq went, you can always argue that things would have been worse. If Saddam Hussein had remained in power. So, you have these counterfactual, you have these factual and counterfactual reference points that people use in debates implicitly to make rhetorical points. So, a lot of people, part of what attracted me to counterfactuals was A, how important they are in drawing any lessons from history, B, how important they are in policy arguments, and C, how unresolvable they are. One of the things I think we're hoping to do in the FOCUS program is to develop some objective metrics for identifying people and methods of generating probabilities that produce superior counterfactual forecasts and simulated worlds in which you can rerun history and assess, you know, what the probability distributions of possible worlds are. Well, it turns out, you know, you get World War I 37% of the time, even if you undo the assassination of the Archduke, and you get something like World War II— you see where we're going. So we're hoping that one result of FOCUS will be to help us identify people and methods that generate superior counterfactual forecasts in domains where there is a ground truth. The next task will be to connect superior performance in simulated worlds to superior performance in the actual world. Now, this is where things get tricky, of course, because in the actual world we don't have these the ground truth. So if I ask you a counterfactual question of the form, you know, if NATO hadn't expanded eastward as far as it did in 2004 into the Baltics, NATO-Russia relations would be considerably friendlier than they are now.

Are chess players good forecasters?

But do you want to go with this for a second?

Sure, I'm sorry.

Yeah, no, no, I don't want to stop I'll stop on that one. No, keep on going. We've got a counterfactual there. Now, we can't rerun history. We don't know how relations with Russia would be if NATO hadn't gone into the Baltics. Right. Russia would have gobbled up the Baltics. Maybe Russia would be friendlier and feel less threatened. You have people who have more hawkish or more dovish mental models of Russia, and those mental models predispose them to give you certain canned, almost ideologically reflexive answers to those counterfactuals. Right? Right. Now, you— so you can— but you can measure what people's beliefs are in the counterfactual, and then you can measure people's beliefs about conditional forecasts that are logically connected to the counterfactuals in a kind of a Bayesian inference network. So, you can measure, you know, what— if I know that you think that the Russians would be every bit as nasty and snarly even if we hadn't moved into the Baltic, they might even be nastier. It's probably a fair bet that you're also likely to think it's a good idea to increase arms sales to the Ukraine, ratchet up sanctions on Putin's cronies, and so forth. So, we can identify the counterfactual belief correlates of more or less accurate conditional forecasting. And in that sense, you can indirectly validate or invalidate— you can render more or less plausible certain counterfactual beliefs. That's the longer-term objective of this research program. It's not just— we're not playing Civilization V for the sake of getting better at Civilization V. The ultimate goal is to link the sophistication of counterfactual reasoning about the past to the subtlety and the accuracy of conditional forecasts going into the future. Another thing you should observe, by the way, if people are becoming better counterfactual reasoners, is you should observe less ideological polarization in their counterfactual beliefs. Yes. So, just the counterfactual belief should become as ideologically depolarized as conditional forecasts are.

Does playing World of Warcraft a lot help you become a better forecaster, or playing chess?

I don't have any evidence bearing on either of those things. But, you know, there's a third variable problem there too.

But you get used to a test, right? You know if you lose. There's very little self-deception. This is true.

I mean, the people who do well in these sorts of things often like games like that.

Venture capitalists, when they try to spot talent in others, do you interpret their behavior in terms of a superforecaster model? Someone like Peter Thiel, he found Mark Zuckerberg, Reid Hoffman, Elon Musk. He's a kind of superforecaster. How do you superforecast talent in other people?

Well, it really helps to be working in an environment in which super bright people are not super rare. It's very, very hard to forecast, to identify talent when the base rate falls below 1 in 1,000, 1 in 10,000, 1 in 100,000. You're looking for a needle in a haystack. But the great advantage that venture capitalists in Silicon Valley have is that the talent pool is relatively rich. So, there's quite a few flaky people that come by seeking their money, for sure, but their odds of success are significantly better than you— than they would be if they're working from a population base rate. And of course, they can tolerate a lot of mistakes because just a few hits will pay for a lot of false positives.

But some do much, much better than others, right? Mike Moritz, Peter Thiel. Right.

The question is, are they doing better because they have better social networks? Are they doing better?

They have better judgment. Do you think superforecasting as a technique also applies to superforecasting how people will do? I'm sorry, I'm— Superforecasting as a technique for predicting the future, does it also apply to predicting how successful people will be?

Um, yes, I, I think everything we know from, um, you know, the overlap between superforecasting and intelligence and the overlap between intelligence and success in various— in many different professional lines of work which suggests that is almost certainly true.

Who first superforecasted your own major success?

I don't know what my first major success was. Some people might challenge whether there has been a major success in my career, but—

Who first saw it coming?

I think probably my advisor at University of British Columbia, Peter Sudfeld, supported me and believed in me when there didn't seem to be very many good reasons for doing so. Kind of a confused Canadian kid who was unsure whether to become a lawyer in Canada or go off to Oxford or go to graduate school in the US, who tipped me toward social science in the US.

And what did he see that other people had not seen? God only knows. Well, if anyone would know, it's you, right? You've lived with it for some time. I think he probably thought—

I thought I was probably bright enough to do well, and he probably thought that I was contrarian and weird enough that there was some possibility of doing something distinctively well, something distinctive and different and doing it well.

Do you think academic advisors in general today undervalue weird students?

I think there's probably a tendency in that direction, yes, because weirdness is a big word. Weirdness takes lots of forms that you and I would— we would not be embarrassed about turning away lots of weird people. Sure.

Do you think people who grow up with the second culture are better forecasters?

Oh, you're thinking of my work with Carmi, Carmi Tadmor? Of course. Yeah. Gosh, you really looked into my vida here. This is highly unusual. I think there does seem to be some advantage there, yeah.

And where does that come from? How does it work?

I think it has to tie— ties into accountability, accountability to conflicting audiences and value pluralism. You have a richer internal dialogue. Uh, you, you learn to balance conflicting perspectives more. You have to be a better perspective taker. Uh, perspective taking is a very important part of superforecasting too.

How do we create more Nate Silvers and Philip Tetlocks? What should we change in the world to get more of you?

Well, I think Nate and I are probably very different creatures.

Um, Sure, but we want more of you both, right? What should we do?

You know, one thing I think that would be useful— I mean, there is this tendency for training in universities to become hyper-professionalized and compartmentalized. So I think it is harder for people who have weird interests that straddle, say, psychology and organizational science and political science and law, history— people who have weird sets of interests, it's hard for them to get traction in the current career environment. Certainly in my home discipline of psychology, I mean, you had this kind of rampant publication inflation in which, you know, PhD students would be very lucky to have to get a job, they'd have to have a ridiculous number of publications, and some of them in A journals. It's the sort of thing we didn't really expect. 30 years ago, we thought, oh, that's a tenure case. Oh, that's a junior hire. So those kinds of pressures, I think, produce a focus, a narrowing of focus. Now you say, well, yes, but you— the great— I said earlier that a lot of the great advances in science come from hedgehogs. We're producing hedgehogs on a on an industrial scale here. And I think there's some advantage that maybe we get a little bit more room for weirdo eclectics.

And the way that departments are carved up, would you change that at all?

You know, a number of academics in the past, like Jim March at UC Irvine many decades ago, tried to do something like that. And Amy Gutmann, the president of the University of Pennsylvania, is trying to do that with, you know, Pan-Integrates Knowledge, which is the the chaired professorships that were used to, among other things, to hire Barb and me and hire a number of other people. And we are sort of floaters. We're not connected to one unit. We have multiple connections. So that's a very hospitable work environment from my point of view. But I don't see a lot of places doing a Jim March UC Irvine experiment. Well, let's integrate all the social sciences together. Amy Gutmann, PIK, Penn Integrates Knowledge kind of program. You don't see too many of them yet. It runs against the grain. I'm not even sure it'll survive at Penn beyond Amy. I mean, the natural tendency will be for departments to want to claw back the resources.

Let's say someone comes up to you and they say, Philip, I would like to be more fox-like. I'm not enough of a fox. What actual advice would you give them to achieve that? What should they do? Or not do. Wake up earlier in the morning, exercise more.

Yeah, how to commit career suicide, become more Fox-like.

Maybe they're not an academic, they're a smart business person. How do they do this?

Uh, I, I, you know, kind of obvious things like read a little bit more outside your field. If you're a liberal, read the Wall Street Journal. If you're a conservative, you read the New New York Times. Expose yourself to distant points of view. Try to cultivate some interests outside your field and try to connect them together. I think there's an optimal distance. I mean, so for history, for example, is— sounds quite very different from what I did. I started as an experimental psychologist, and history looks very different. But they can be connected because historical judgment is something that psychologists study to some degree. Psychologists are interested in hindsight and counterfactuals and so forth. So, you can link the two. So, I think there's an optimal distance. And when you go foraging as a fox, you probably don't wanna forage way, way far away. You wanna forage far enough away that'll be stimulating, but still possible to reconnect.

What kinds of people are best at adversarial collaboration?

Rare people. Those are rare people also.

Rare people. But how do you spot them?

Really hard to know. Do.

Is it a personality trait, a cognitive trait?

Gosh, that's a hard one. I mean, that was a thing that Danny Kahneman, I think, coined the term when he was, you know, dealing with his various critics over time. And my wife Barb actually was involved in an adversarial collaboration between the Kahneman camp and the Gigerenzer camp on the conjunction fallacy. It took at least 2 or 3 years of her life. It's a hard It's very hard to get people to— well, I mean, lots of us in principle are Popperians. We believe in stating our beliefs as falsifiable hypotheses. And we also— most of us believe our beliefs are probabilistic. So we're kind of somewhat Bayesian. But that's lip service. There's what we believe. There's our formal set of opinions. Our epistemological formal self-concept, which is kind of noble and falsificationist and probabilistic. And then there's how we actually behave when our egos are at stake in particular controversies. And those things are quite different. I think Danny Kahneman, who's not known as an optimist, you know, did propose— it sounds like an optimistic idea, but I think he's not all that optimistic optimistic about what it can achieve on close inspection. My efforts at adversarial collaboration have not been all that successful. I'd like to jumpstart a few of them again, but it's very hard to find the right dance partners.

Let's try a question from the realm of the everyday and the mundane. If I go around and I look at Mexican restaurants, I'm very good at predicting which ones have excellent tacos. What are you good at predicting?

I'm not good at predicting your questions. What am I good at predicting? I think I was pretty good at anticipating the fragility of a lot of microsocial science knowledge prior to the replication crisis erupting.

No, I mean everyday life. Oh, okay, so not, not, not, not in my— not social events, not social science. What in your life are you good at predicting? When you're going to get tired and want to go to bed at night, or when the dog wants to eat?

What is it? Well, we have a pet-free existence, and, um, you know, we don't— our lives are actually pretty, pretty, pretty simple. I guess we subscribe to that old adage, you should be boring and staid in your life so you can be violent and creative in your work. We have a kind of a routine here.

So it's highly predictable.

And now, you know, and then in the quarantine days, I mean, it used to be pre-quarantine, you know, we said, oh, you know, let's, let's, we're going to go to Europe, we're going to go here or there. There were these little points of unpredictability that spiced up life, but those do not exist right now.

And two more questions to close. First, what can you tell us about your next project?

Well, I think the next project is the one that I mentioned earlier. It's linking historical counterfactual reasoning with conditional forecasting. I think it'll be the second phase of the FOCUS research tournaments I think that counterfactual reasoning has for too long been the last refuge of ideological scoundrels. And it'll be— insofar as we can improve the standards of evidence and proof in judging counterfactual claims as well as conditional forecasts and linking the two, I think there's a potential for improving the quality of debates among interested parties.

And finally, what should a superforecaster predict? About the future course of your influence?

Oh, to be very cautious because, you know, we're running against the grain. We're running against the psychological grain, we're running against human nature, we're running against the sociological grain.

But the Enlightenment is stable, you tell us, right? So if it keeps on accumulating and growing, your influence should be enormous. That's on your other presuppositions.

Well, there, there, there's a lot of— there's a lot of cognitive resistance to treating one's beliefs as false falsifiable probabilistic propositions. People naturally gravitate toward thinking of their beliefs as ego-defining, quasi-sacred possessions. And that's one major source of— that's one major obstacle. Then you have the existing status hierarchies. You have subject matter experts who are entrenched, have influence. Why would they want to participate in exercises in which the best possible outcome is a tie, which they reaffirm that they deserve the status that they already have? So that's not psychological, sociological resistance. You know, it's interesting, you know, the sociologists and economists have kind of different reactions to forecasting tournaments. The sociologists, they react, you know, why would, why would anyone be naive enough to think that anyone would want to have a forecasting tournament in their organization? They're status disruptive, right? Yes. And economists would say, well, if these things are so great, how come they're not everywhere?

And with that, Philip Tetlock, thank you very much.

It's been a pleasure. Take care.