David Ferrucci: AI Understanding the World Through Shared Knowledge Frameworks | AI Podcast Clips

Do you think that shared knowledge, if we can maybe escape the hardware question, how much is encoded in the hardware, just the shared knowledge in the software, the history, the many centuries of wars and so on that came to today, that shared knowledge, how hard is it to encode? Do you have a hope? Can you speak to how hard is it to encode that knowledge systematically in a way that could be used by a computer?

So I think it is possible to learn, for machine, to program a machine to acquire that knowledge with a similar foundation, in other words, a similar interpretive foundation for processing that knowledge.

What do you mean by that?

So, in other words, we view the world in a particular way. So in other words, we have a, if you will, as humans, we have a framework for interpreting the world around us. So we have multiple frameworks for interpreting the world around us. But if you're interpreting, for example, sociopolitical interactions, you're thinking about whether there's people, there's collections and groups of people, they have goals. Goals are largely built around survival and quality of life. There are. There are fundamental economics around scarcity of resources. And when humans come and start interpreting a situation like that, because you brought up, like historical events, they start interpreting situations like that. They apply a lot of this fundamental framework for interpreting that, well, who are the people? What were their goals? What resources did they have? How much power, influence did they have over the other? Like, there's fundamental substrate, if you will, for interpreting and reasoning about that. So I think it is possible to imbue a computer with that stuff that humans take for granted when they go and sit down and try to interpret things. And then with that foundation, they acquire, they start acquiring the details, the specifics in a given situation are then able to interpret it with regard to that framework. And then given that interpretation, they can do what they can predict. But not only can they predict, they can predict now with an explanation that can be given in those terms, in the terms of that underlying framework that most humans share. Now, you could find humans that come and interpret events very differently than other humans because they're using a different framework. The movie Matrix comes to mind where they decided humans really just batteries, and that's how they interpreted the value of humans as a source of electrical energy. But I think that for the most part, we have a way of interpreting the events or the social events around us because we have this shared framework. It comes from, again, the fact that we're similar beings that have similar goals, similar emotions, and we can make sense out of these. These frameworks make Sense to us.

So how much knowledge is there, do you think? So you said it's possible.

Well, there's a tremendous amount of detailed knowledge in the world. You can imagine effectively infinite number of unique situations and unique configurations of these things. But the knowledge that you need, what I refer to as like the frameworks for, you need for interpreting them, I don't think. I think those are finite.

You think the frameworks are more important than the bulk of the knowledge. So like framing.

Yeah, because what the frameworks do is they give you now the ability to interpret and reason and to interpret and reason. To interpret and reason over the specifics in ways that other humans would understand.

What about the specifics?

You acquired the specifics by reading and by talking to other people.

So mostly actually just even if you can focus on even the beginning, the common sense stuff, the stuff that doesn't even require reading or it almost requires playing around with the world or something, just being able to sort of manipulate objects, drink water and so on, all that. Every time we try to do that kind of thing in robotics or AI, it seems to be like an onion. You seem to realize how much knowledge is really required to perform even some of these basic tasks. Do you have that sense as well? And if so, how do we get all those details? Are they written down somewhere? Do they have to be learned through experience?

So I think when, like, if you're talking about sort of the physics, the basic physics around us, for example, acquiring information about acquiring how that works. Yeah, I think there's a combination of things going. I think there's a combination of things going on. I think there is fundamental pattern matching, like what we were talking about before, where you see enough examples, enough data about something, you start assuming that. And with similar input, I'm going to predict similar outputs. You can't necessarily explain it at all. You may learn very quickly that when you let something go, it falls to the ground.

That's such a.

You can't necessarily explain that.

But that's such a deep idea that if you let something go, like the idea of gravity, I mean, people were.

Letting things go and counting on them falling well before they understood gravity.

But that seems to be. That's exactly what I mean is before you take a physics class or, or study anything about Newton, just the idea that stuff falls to the ground and then you be able to generalize that all kinds of stuff falls to the ground, it just seems like a non without encoding it, like hard coding it in. It seems like a difficult thing to pick up. It Seems like you have to have a lot of different knowledge to be able to integrate that into the framework, sort of into everything else. So both know that stuff falls to the ground and start to reason about socio political discourse. So both like the very basic and the high level reasoning decision making. I guess my question is how hard is this problem? And sorry to linger on it because again, and we'll get to it for sure as what Watson with Jeopardy. Did is take on a problem that's much more constrained but has the same hugeness of scale, at least from the outsider's perspective. So I'm asking the general life question of to be able to be an intelligent being and reasoning in the world about both gravity and politics. How hard is that problem?

So I think it's solvable.

That's okay now. Beautiful. So what about, what about time travel? Okay, not as convinced, not as convinced yet. Okay.

No, I think it is solvable. I mean, I think that first of all it's about getting machines to learn. Learning is fundamental and I think we're already in a place that we understand, for example, how machines can learn in various ways. Right now our learning stuff is sort of primitive in that we haven't sort of taught machines to learn the frameworks. We don't communicate our frameworks because of how shared. In some cases we do, but we don't annotate, if you will, all the data in the world with the frameworks that are inherent or underlying our understanding. Instead, we just operate with the data. So if we want to be able to reason over the data in similar terms in the common frameworks, we need to be able to teach the computer, or at least we need to program the computer to acquire, to have access to, and acquire, learn the frameworks as well and connect the frameworks to the data. I think this, I think this can be done. I think we can start, I think machine learning, for example, with enough examples can start to learn these basic dynamics. Will they relate them necessarily to gravity? Not unless they can also acquire those theories as well and put the experiential knowledge and connect it back to the theoretical knowledge. I think if we think in terms of these class of architectures that are designed to both learn the specifics, find the patterns, but also acquire the frameworks and, and connect the data to the frameworks. If we think in terms of robust architectures like this, I think there is a path toward getting there.

In terms of encoding architectures like that, do you think systems that were able to do this will look like neural networks or representing if you look back to the 80s and 90s with the expert systems, so they're more like graphs, systems that are based in logic, able to contain a large amount of knowledge, where the challenge was the automated acquisition of that knowledge. I guess the question is, when you collect both the frameworks and the knowledge from the data, what do you think that thing will look like?

Yeah, so I mean, I think asking the question they look like neural networks is a bit of a red herring. I mean, I think that they will certainly do inductive or pattern match based reasoning. And I already experimented with architectures that combine both that use machine learning and neural networks to learn certain classes of knowledge. In other words, to find repeated patterns in order for it to make good inductive guesses, but then ultimately to try to take those learnings and marry them. In other words, connect them to frameworks so that it can then reason over that in terms other humans understand. So for example, at elemental cognition, we do both. We have architectures that do both those things, but also have a learning method for acquiring the frameworks themselves and saying, look, ultimately I need to take this data, I need to interpret it in the form of these frameworks so they can reason over it. So there is a fundamental knowledge representation, like what you're saying, like these graphs of logic, if you will. There are also neural networks that acquire a certain class of information, then they then align them with these frameworks, but there's also a mechanism to acquire the frameworks themselves. Yeah.

So it seems like that idea of frameworks requires some kind of collaboration with humans.

Absolutely.

So do you think of that collaboration as well?

And let's be clear, only for the express purpose that you're designing, machine designing and intelligence that can ultimately communicate with humans in, in the terms of frameworks that help them understand things. Right. So, so now to be really clear, you can create, you can independently create an, a machine learning system and an intel. An intelligence that I might call an alien intelligence that does a better job than you with some things, but can't explain the framework to you. That doesn't mean it might be better than you at the thing. It might be that you cannot comprehend the framework that it may have created for itself. That is inexplicable to you, that's a reality.

But you're more interested in a case where you can.

I am, yeah. My sort of approach to AI is because I've set the goal for myself. I want machines to be able to ultimately communicate understanding with humans. I want them to be able to acquire and communicate. Acquire knowledge from humans and communicate knowledge to humans. They should be using what inductive machine learning techniques are good at, which is to observe patterns of data, whether it be in language or whether it be in images or videos or whatever, to acquire these patterns, to induce the generalizations from those patterns, but then ultimately to work with humans to connect them to frameworks, interpretations, if you will, that ultimately make sense to humans. Of course, the machine is going to have the strength, it has the richer, longer memory, but it has the more rigorous reasoning abilities, the deeper reasoning abilities. So it'll be an interesting complementary relationship between the human and the machine.

Do you think that ultimately needs explainability, like a machine? So if we look, we study, for example, Tesla Autopilot, a lot where humans, I don't know if you've driven the vehicle or are aware of what. So you're basically the human and machine are working together there, and the human is responsible for their own life to monitor the system. And, you know, the system fails every few miles. And so there's. There's hundreds of. There's millions of those failures a day. And so that's like a moment of interaction. Do you see?

Yeah, no, that's exactly right. That's a moment of interaction where, you know, the machine has learned some stuff, it has a failure. Somehow the failure is communicated. The human is now filling in the mistake, if you will, or maybe correcting or doing something that is more successful. In that case, the computer takes that learning. So I believe that the collaboration between human and machine, I mean, that's sort of a primitive example and sort of a. More. Another example is where the machine's literally talking to you and saying, look, I'm reading this thing. I know that the next word might be this or that, but I don't really understand why. I have my guess. Can you help me understand the framework that supports this and then can kind of acquire that, take that and reason about it and reuse it the next time it's reading to try to understand something not unlike a human student might do? I mean, I remember when my daughter was in first grade and she had a reading assignment about electricity. And somewhere in the text it says, and electricity is produced by water flowing over turbines or something like that. And then there's a question that says, well, how is electricity created? And so my daughter comes to me and says, I mean, I could, you know, created and produced are kind of synonyms in this case. So I can go back to the text and I can copy by water flowing over turbines. But I have no idea what that means. Like, I don't know how to interpret water flowing over turbines and what electricity even is. I mean, I can get the answer right by matching the text, but I don't have any framework for understanding what this means at all.

And framework, really? I mean, it's a set of. Not to be mathematical, but axioms of ideas that you bring to the table in interpreting stuff, and then you build those up somehow.

You build them up with the expectation that there's a shared understanding of what they are.

Sure, yeah. It's the social that us humans do. You have a sense that humans on Earth in general share a set of, like, how many frameworks are there?

I mean, it depends on how you bound them. Right. So, in other words, how big or small, like their individual scope. But there's lots and there are new ones. I think the way I think about is kind of in a layer. I think of the architectures being layered in that there's a small set of primitives that allow you the foundation to build frameworks. And then there may be many frameworks, but you have the ability to acquire them and. And then you have the ability to reuse them. I mean, one of the most compelling ways of thinking about this is reasoning by analogy, where I can say, oh, wow, I've learned something very similar. I never heard of this game soccer, but if it's like basketball in the sense that the goal is like the hoop, and I have to get the ball and the hoop and I have guards and I have this and I have that, where are the similarities and where are the differences? And I have a foundation now for interpreting this new information.

And then different groups like the millennials will have a framework and then.

Well, that, you know. Yeah, well, like that.

Democrats and Republicans, Millennials. Nobody wants that framework.

Well, I mean, I think nobody understands it. Right. You're talking about political and social ways of interpreting the world around them. And I think these frameworks are still largely, largely similar. I think they differ in maybe what some fundamental assumptions and values are. But now, from a reasoning perspective, like the ability to process the framework might not be that different. The implications of different fundamental values or fundamental assumptions in those frameworks may reach very different conclusions. So from a social perspective, the conclusions may be very different. From an intelligence perspective, I just followed where my assumptions took me.

Yeah, the process itself will look similar, but that's a fascinating idea that frameworks really help carve how a statement will be interpreted. I mean, having a Democrat and a Republican framework and read the Exact same statement. And the conclusions that you derive will be totally different from an AI perspective is fascinating.

What we would want out of the AI is to be able to tell you that this perspective, one perspective, one set of assumptions, is going to lead you here, another set of assumptions is going to lead you there. And in fact, to help people reason and say, oh, I see where our differences lie. I have this fundamental belief about that. I have this fundamental belief about that.

Yeah, that's quite brilliant from my perspective. Nlp, there's this idea that there's one way to really understand a statement, but there probably isn't. There's probably an infinite number of ways to understand.

Well, there's lots depending on the lots of different interpretations and the broader the content, the richer it is. And so you and I can have very different experiences with the same text, obviously. And if we're committed to understanding each other, we start. And that's the other important point. Like if we're committed to understanding each other, we start decomposing and breaking down our interpretation towards more and more primitive components until we get to that point where we say, oh, I see why we disagree. And we try to understand how fundamental that disagreement really is. But that requires a commitment to breaking down that interpretation in terms of that framework in a logical way. Otherwise. And this is why I think of AI as really complementing and helping human intelligence to overcome some of its biases and its predisposition to be persuaded by more shallow reasoning in the sense that we get over this idea. Well, I'm right because I'm Republican, or I'm right because I'm Democratic. And someone labeled this as a Democratic point of view, or it has the following keywords in it. And if the machine can help us break that argument down and say, wait a second, what do you really think about this? Right. So essentially holding us accountable to doing more critical thinking.

We're going to have to sit and think about that. I love that. I think that's really empowering use of AI for the public discourse that's completely disintegrating currently as we learn how to do it on social media.

That's right, That.