David Ferrucci: The Story of IBM Watson Winning in Jeopardy | AI Podcast Clips

So one of the greatest accomplishments in the history of AI Is Watson competing against, in the game of Jeopardy. Against humans. And you were a lead in that. A critical part of that. Let's start at the very basics. What is the game of Jeopardy. The game for us humans, human versus human.

Right. So it's to take a question and answer it. Well, it's just the opposite, actually.

It's the opposite.

Well, no, but it's not right. It's really not. It's really, it's really to get a question and answer, but it's what we call a factoid question. So this notion of like, it really relates to some fact that few people would argue whether the facts are true or not. In fact, most people what in Jeopardy kind of counts on the idea that these, these statements have factual answers. And the idea is to first of all determine whether or not you know the answer, which is sort of an interesting twist.

So first of all, understand the question.

You have to understand the question. What is it asking? And that's a good point because the questions are not asked directly. Right.

They're all like, the way the questions are asked is non linear. It's like, it's a little bit witty, it's a little bit playful sometimes. It's. It's a little bit tricky.

Yeah, they're asked in exactly numerous witty, tricky ways. Exactly. What they're asking is not obvious. It takes inexperienced humans a while to go, what is it even asking?

Right.

And that's sort of an interesting realization that you have when somebody says, oh, what's the Jeopardy Is a question answering show. And then he's like, oh, like I know a lot. And then you read it and you're still trying to process the question. And the champions have answered and moved on. There are three questions ahead by the time you figured out what the question even meant. So there's, there's definitely an ability there to just parse out what the question even is. So that was certainly challenging. It's interesting historically though, if you look back at the Jeopardy. Games much earlier, you know, like 60 70s, the questions were much more direct. They weren't quite like that. They got sort of more and more interesting the way they asked them. That sort of got more and more interesting and subtle and nuanced and humorous and witty over time, which really required the human to kind of make the right connections in figuring out what the question was even asking. So, yeah, you have to figure out the questions even asking. Then you have to determine whether or not you think you know the answer. And because you have to buzz in really quickly, you sort of have to make that determination as quickly as you possibly can. Otherwise you lose the opportunity to buzz.

In even before you really know if you know the answer.

I think a lot. I think a lot of humans will. Will assume they'll, they'll look at. They'll look at it. They process it very superficially. In other words, what's the topic? What are some keywords? And just say, do I know or not? Before they actually know the answer, then they'll buzz in, and then they'll buzz in and think about it. So it's interesting what humans do. Now. Some people who know all things, like Ken Jennings or something or the more recent big Jeopardy. Player, I mean, they'll just buzz it. They'll just assume they know all of Jeopardy. And they'll just post it. You know, Watson, interestingly, didn't even come close to knowing all of Jeopardy. Right, Watson.

Even at the peak, Even at its best.

Yeah. So, for example, I mean, we had this thing called recall, which is like, how many of all the Jeopardy. Questions, you know, how many did could we even find, like, find the right answer for, like, anywhere? Like, could we come up with if we look, you know, we had a, a big body of knowledge, something in the order of several terabytes. I mean, from, from a web scale was actually very small. But from like a book scale, talking about millions of books. Right. So the equivalent millions of books, encyclopedias, dictionaries, books, it's still a ton of information. And I think it was only 85% was the answer anywhere to be found. So you're already down at that level just to get started. And so it was important to get a very quick sense of, do you think you know the right answer to this question? So we had to compute that confidence as quickly as we possibly could. So in effect, we had to answer it and at least spend some time essentially answering it and then judging the confidence that we, you know, that, that our answer was right, and then deciding whether or not we were confident enough to buzz in, and that would depend on what else was going on in the game because it was a risk. So, like, if you're really in a situation where I have to take a guess, I have very little to lose, then you'll buzz in with less confidence.

So that was accounted for, the financial standings of the different competitors.

Correct. How much of the game was laugh. How much time was left where you were in the standing and things like that.

What, how many hundreds of milliseconds that we're talking about here. Do you have a sense of what is we targeted?

What's the targeted. So I mean we targeted answering in under three seconds and buzzing it.

So the decision to buzz in and then the actual answering are those two?

Yeah, they were two, they were two different things. In fact we had multiple stages. Whereas like we, let's estimate our confidence, which is sort of a shallow answering process and then ultimately decide to buzz in and then we may take another second or something to kind of go in there and do that. But by and large we're saying like we can't play the game. We can't even compete if we can't on average answer these questions in around three seconds or less.

So you stepped in. So there's these three humans playing a game and you stepped in with the idea that IBM Watson would be one of replace one of the humans and compete against two. Can you tell the story of Watson taking on this game?

Sure.

Seems exceptionally difficult.

Yeah. So the story was that it was coming up, I think to the 10 year anniversary of Big Blue, not Big Blue, Deep Blue. IBM wanted to do sort of another kind of really, you know, fun challenge, public challenge that can bring attention to IBM research and the kind of the cool stuff that we were doing. I had been working in AI at IBM for some time. I had a team doing what's called open domain factoid question answering, which is, you know, we're not going to tell you what the questions are, we're not even going to tell you what they're about. Can you go off and get accurate answers to these questions? And it was an area of AI research that I was involved in. And so it was a very specific passion of mine. Language understanding had always been a passion of mine. One sort of narrow slice on whether or not you could do anything with language was this notion of open domain, meaning I could ask anything about anything. Factoid meaning it essentially had an answer and being able to do that accurately and quickly. So that was a research area that my team had already been in. And so completely independently, several IBM executives like what are we going to do? What's the next cool thing to do? And. And Ken Jennings was on his winning streak. This was like whatever it was 2004, I think was on his winning streak and someone thought, hey, that'll be really cool if the computer can play Jeopardy. And so this was like in 2004, they were shopping this thing around and everyone was telling the research execs, no way, like this is crazy. And we have some Pretty senior people in the field and they're saying, this is crazy and it'll come across my desk. And I was like, but that's kind of what I'm really interested in doing. But there was such, this prevailing sense of this is nuts, we're not going to risk IBM's reputation on this. We're just not doing it. And this happened in 2004, it happened in 2005. At the end of 2006, it was coming around again and I was coming off of a. I was doing the open domain question answering stuff, but I was coming off a couple other projects. I had a lot more time to put into this and I argued that it could be done and I argue it would be crazy not to do this.

Can I. You could be honest at this point. So even though you argued for it, what's the confidence that you had yourself privately that this could be done was we just told the story how you tell stories to convince others. How confident were you? What was your estimation of the problem at that time?

So I thought it was possible. And a lot of people thought it was impossible. I thought it was possible. The reason why I thought it was possible is because I did some brief experimentation. I knew a lot about how we were approaching open domain factoid question answering. We've been doing it for some years. I looked at the Jeopardy stuff. I said, this is going to be hard for a lot of the points that we mentioned earlier. Hard to interpret the question, hard to do it quickly enough, hard to compute an accurate confidence. None of this stuff had been done well enough before. But a lot of the technologies we're building were the kinds of technologies that should work. But more to the point, what was driving me was I was in IBM research. I was a senior leader in IBM research. And this is the kind of stuff we were supposed to do. In other words, we were basically supposed to.

This is the moonshot.

This is the. I mean, we were supposed to take things and say, this is an active research area. It's our obligation to kind of, if we have the opportunity to push it to the limits and if it doesn't work, to understand more deeply why we can't do it. And so I was very committed to that notion, saying, folks, this is what we do. It's crazy not to do this. This is an active research area. We've been in this for years. Why wouldn't we take this grand challenge and push it as hard as we can? At the very least, we'd be able to come out and say, here's why this problem is way hard. Here's what we tried and here's how we fail. So I was very driven as a scientist from that perspective. And then I also argued, based on what we did, a feasibility study, why I thought it was hard, but possible. And I showed examples of where it succeeded, where it failed, why it failed, and sort of a high level architectural approach for why we should do it. But for the most part, at that point, the execs really were just looking for someone crazy enough to say yes. Because for several years at that point everyone had said, no, I'm not willing to risk my reputation and my career, you know, on this thing.

Clearly you did not have such fears.

Okay, I did not.

So you dived right in. And yet, for what I understand, it was performing very poorly in the beginning. So what were the initial approaches and why did they fail?

Well, there were lots of hard aspects to it. I mean, one of the reasons why prior approaches that we had worked on in the past failed was because of, because the questions were difficult, difficult to interpret. Like what are you even asking for? Right? Very often, like if, if the question was very direct, like what city you know, or what you know, even then it could be tricky. But, but you know, what city or, or what person. Often when it would name it very clearly, you would know that. And if there were just a small set of them. In other words, we're going to ask about these five types, like it's going to be an answer, and the answer will be a city in this state or a city in this country. The answer will be a person of this type. Right? Like an actor or whatever it is. But it turns out that in jeopardy there were like tens of thousands of these things and it was a very, very long tail, meaning, you know, it just went on and on. And so even if you focused on trying to encode the types at the very top, like there's five that were the most, let's say five of the most frequent, you still cover a very small percentage of the data. So you couldn't take that approach of saying, I'm just going to try to collect facts about these 5 or 10 types or 20 types or 50 types or whatever. So that was one of the first things like what do you do about that? And so we came up with an approach toward that and the approach looked promising and we continue to improve our ability to handle that problem throughout the project. The other issue was that right from the outset I said we're not going to, I committed to doing this in three to five years. So we did it in four. So I got lucky. But one of the things that, putting that stake in the ground and I knew how hard the language understanding problem was, I said we're not going to actually understand language to solve its problem. We are not going to interpret the question and the domain of knowledge the question refers to in reason over that to answer these questions. Obviously we're not going to be doing that. At the same time, simple search wasn't good enough to confidently answer with a single correct answer.

First of all, that's brilliant. That's such a great mix of innovation and practical engineering. 3, 3, 4, 8. So you're not, you're not trying to solve the general NLU problem. You're saying let's solve this in any way possible.

Oh yeah, no, I was committed to saying, look, we're solving the open domain question answering problem. We're using Jeopardy as a driver for that big benchmark. Hard enough. Big benchmark. Exactly. And now we're, how do we do it? We could just like, whatever, like just figure out what works. Because I want to be able to go back to the academic science community and say, here's what we tried, here's what worked, here's what didn't work great. I don't want to go in and say, oh, I only have one technology, I have a hammer, I'm only going to use this, I'm going to do whatever it takes. I'm like, I'm going to think out of the box and do whatever it takes. 1. And I also, there was another thing I believed, I believed that the fundamental NLP technologies and machine learning technologies would be adequate. And this was an issue of how do we enhance them, how do we integrate them, how do we advance them? So I had one researcher and came to me who had been working on question answering with me for a very long time who had said we're going to need Maxwell's equations for question answering. And I said if we need some fundamental formula that breaks new ground and how we understand language, we're screwed. We're not going to get there from here. I am not counting, my assumption is I'm not counting on some brand new invention. What I'm counting on is the ability to take everything it has done before to figure out an architecture on how to integrate it well and then see where it breaks and make the necessary advances we need to make until this thing works.

Yeah, push it hard to see where it breaks and then patch it up. I mean that's how People change the world. And that's the Elon Musk approach with rockets. SpaceX, that's the Henry Ford and so on.

I love that. And I happen to be, and in this case I happen to be right, but like we didn't know. But you kind of have to put a stake in the ground. How are you going to run the project?

So, yeah, and backtracking to search. So if you were to do what's the brute force solution, what would you search over? So you have a question, how would you search the possible space of answers?

Look, web search has come a long way even since then. But at the time, first of all, there are a couple of other constraints around the problem, which is interesting. So you couldn't go out to the web, you couldn't search the Internet. In other words, the AI experiment was we want a self contained device. If device is as big as a room, fine, it's as big as a room. But we want a self contained device, contained device. You're not going out the Internet, you don't have a lifeline to anything. So it had to kind of fit in a shoebox, if you will, or at least size of a few refrigerators, whatever it might be. But also you couldn't just get out there, you couldn't go off network to kind of go. So there was that limitation. But the basic thing was go do a web search. The problem was, even when we went and did a web search, I don't remember exactly the numbers, but somewhere in the order of 65% of the time, the answer would be somewhere in the top 10 or 20 documents. So first of all, that's not even good enough to play Jeopardy. In other words, even if you could pull the, even if you could perfectly pull the answer out of the top 20 documents, top 10 documents, whatever it was, which we didn't know how to do. But even if you could do that and you knew it was right, you had enough confidence in it, you'd have to pull out the right answer, you'd have to have confidence it was the right answer. And then you'd have to do that fast enough to now go buzz in and you'd still only get 65% of them right, which doesn't even put you in the winner circle. Winner circle. You have to be up over 70 and you have to do it really. And you have to do it really quickly. But now the problem is, well, even if I had somewhere in the top 10 documents, how do I figure out where in the top 10 documents that answer is? And how do I compute a confidence of all the possible candidates? So it's not like I go in knowing the right answer and have to pick it. I don't know the right answer. I have a bunch of documents. Somewhere in there is the right answer. How do I as a machine go out and figure out which one's right and then how do I score it? So and now how do I deal with the fact that I can't actually go out to the web?

First of all, if you pause on that, just think about it. If you could go to the web, do you think that problem is solvable if you just pause on it? Just thinking even beyond Jeopardy, do you think the problem of reading text to find where the answer is?

Well, we saw we solved that in some definition of solved. Given the Jeopardy. Challenge.

How did you do it for Jeopardy? So how do you take a body of work in a particular topic and extract the key pieces of information?

So what? So now forgetting about the huge volumes that are on the web, right? So now we have to figure out, we did a lot of source research. In other words, what body of knowledge is going to be small enough but broad enough to answer Jeopardy? And we ultimately did find the body of knowledge that did that. I mean, it included Wikipedia and a bunch of other stuff.

So like encyclopedia type of stuff. I don't know if you.

Encyclopedias, dictionaries, different types of semantic resources like WordNet and other types of semantic resources like that, as well as like some web crawls, in other words, where we went out and took that content and then expanded it based on producing statistical seed, you know, statistically producing seeds, using those seeds for other searches, searches and then expanding that. So using these like expansion techniques, we went out and had found enough content that we're like, okay, this is good. And even up until the end, you know, we had a thread of researchers always trying to figure out what content could we efficiently include.

I mean, there's a lot of popular. Like what is the church lady? Well, I think was one of the. Like, yeah, what. Where do you get. I guess that's probably an encyclopedia.

So that's encyclopedia. But then we would take that stuff and we would go out and we would expand. In other words, we go find other content that wasn't in the core resources and expand it the amount of content, grew it by an order of magnitude. But still, again, from a web scale perspective, this is very small amount of content.

It's very select.

We then took all that content, we pre analyzed the crap out of it, meaning we parsed it, broke it down into all its individual words, and then we did statistic and semantic parses on it, had computer algorithms that annotated it and we indexed that in a very rich and very fast index. So we have a relatively huge amount of, let's say the equivalent of, for the sake of argument, two to five million bucks. We've now analyzed all that, blowing up its size even more because now we have all this metadata and, and we then we richly indexed all of that and by the way, in a giant in memory cache. So Watson did not go to disk.

So the infrastructure component there, if you could just speak to it, how tough? I mean, I know 2000, maybe this is 2008, 9. You know that that's kind of a long time ago, right? How hard is it to use multiple machines? Like how hard is the infrastructure component, the hardware component?

So we used IBM, so we used IBM hardware. We had something like, I forget exactly. But 2000, close to 3000 cores completely connected. So you had a switch where, you know, every CPU was connected to every.

Other CPU and they were sharing memory in some kind of way, kind of.

Clever shared memory, right? And all this data was pre analyzed and put into a very fast indexing structure that was all in memory. And then we took that question, we would analyze the question. So all the content was now pre analyzed. So if I went and tried to find a piece of content, it would come back with all the metadata that we had pre computed.

How do you shove that question? How do you connect the big stuff with the big knowledge base with the metadata? And that's indexed to the simple little witty, confusing question, right?

So therein lies the Watson architecture. So we would take the question, we would analyze the question, which means that we would parse it and interpret it a bunch of different ways. We'd try to figure out what is it asking about. So we had multiple strategies to kind of determine what was it asking for that might be represented as a simple string, a character string or something. We would connect back to different semantic types that were from existing resources. So anyway, the bottom line is we would do a bunch of analysis in the question and question analysis had to finish and had to finish fast. So we do the question analysis because then from the question analysis we would now produce searches and we had built using open source search engines. We modified them, but we had a number of different search engines we would use that had different characteristics. We went in there and engineered and modified those search engines ultimately to now take our question analysis, produce multiple queries based on different interpretations of the question and fire out a whole bunch of searches in parallel. And they would come back with passages. So these are passage search algorithms. They would come back with passages. And so now let's say you had 1,000 passages. Now for each passage, you parallelize again. So you went out and you paralyzed the search. Each search would now come back with a whole bunch of passages. Maybe you had a total of 1,000 or 5,000, whatever passages for each passage. Now you'd go and figure out whether or not there was a candidate, we'd call a candidate answer in there. So you had a whole bunch of other algorithms that would find candidate answers, possible answers to the question. And so you had candidate answers called candidate answers generators, a whole bunch of those. So for every one of these components, the team was constantly doing research, coming up better ways to generate search queries from the questions, better ways to analyze the question, better ways to generate candidates, and speed.

So better is accuracy and speed, correct?

So, right. Speed and accuracy for the most part were separated. We handle that sort of in separate ways. Like I focus purely on accuracy, end to end accuracy. Are we ultimately getting more questions and producing more accurate confidences? And then a whole other team that was constantly analyzing the workflow to find the bottlenecks and then figuring out how to both parallelize and drive the algorithm speed. But anyway, so now think of it like you have this big fan out now, right? Because you have, you had multiple queries. Now you have, now you have thousands of candidate answers. For each candidate answer, you're going to score it. So you're going to use all the data that built up, you're going to use the question analysis, you're going to use how the query was generated, you're going to use the passage itself, and you're going to use the candidate answer that was generated and you're going to score that. So now we have a group of researchers coming up with scorers. There are hundreds of different scorers. So now you're getting a fan at it again from however many candidate answers you have to all the different scores. So if you have 200 different scores and you have a thousand candidates, now you have 200,000 scores. And so now you got to figure out, how do I now rank these answers based on the scores that came back. And I want to rank them based on the likelihood that they're a correct answer to the question. So every score was its own research project.

What do you mean by score? So is that the annotation process of basically a human being.