Yann LeCun: Benchmarks for Human-Level Intelligence | AI Podcast Clips

Foreign. You've written advice saying, don't get fooled by people who claim to have a solution to artificial general intelligence, who claim to have an AI system that work just like the human brain, or who claim to have figured out how the brain works. Ask them what the error rate they get on MNIST or imagenet.

This is a little dated, by the way.

I mean, five years, who's counting? Okay, but I think your opinion is still MNIST and imagenet. Yes, may be data, there may be new benchmarks. Right. But I think that philosophy is one you still somewhat hold that benchmarks and the practical testing, the practical application is where you really get to test the ideas.

Well, it may not be completely practical, like, for example, it could be a toy data set, but it has to be some sort of task that the community as a whole has accepted as some sort of standard kind of benchmark. If you want. It doesn't need to be real. So, for example, many years ago here at Fair, Jason Weston, Otwain Bourne and a few others proposed the baby tasks, which were kind of a toy problem, to test the ability of machines to reason, actually to access working memory and things like this. And. And it was very useful, even though it wasn't a real task. Mnist is kind of halfway a real task. So toy problems can be very useful. It's just that I was really struck by the fact that a lot of people, particularly a lot of people with money to invest, would be fooled by people telling them, oh, we have the algorithm of the Cortex and you should give us 50 million.

Yes, absolutely. So there's a lot of people who, who try to take advantage of the hype for business reasons and so on. But let me sort of talk to this idea that new ideas, the ideas that push the field forward may not yet have a benchmark or it may be very difficult to establish a benchmark.

I agree. That's part of the process. Establishing benchmarks is part of the process.

So what are your thoughts about. So we have these benchmarks on. Around stuff we can do with images, from classification to captioning, to just every kind of information you can pull off from images and the surface level. There's audio data sets, there's some video. What can we start? Natural language. What kind of stuff? What kind of benchmarks do you see that start creeping on to more something like intelligence, like reasoning, like maybe you don't like the term, but AGI echoes of that kind of simulation.

A lot of people are working on interactive environments in which you can train and test intelligent Systems. For example, the classical paradigm of supervised learning is that you have a data set, you partition it into a training set, validation set, test set, and there's a clear protocol. But what if that assumes that the samples are statistically independent? You can exchange them. The order in which you see them shouldn't matter, things like that. But what if the answer you give determines the next sample you see, which is the case, for example, in robotics, a robot does something and then it gets exposed to a new room. And depending on where it goes, the room will be different. So that creates the exploration problem, the what if the samples. So that creates also a dependency between samples. Right. If you can only move in space, the next sample you're going to see is going to be probably in the same building, most likely. So all the assumptions about the validity of this training set, test set, hypothesis break. Whenever a machine can take an action that has an influence in the world, and it's what it's going to see. So people are setting up artificial environments where that takes place. Right. The robot runs around a 3D model of a house and can interact with objects and things like this. You do robotics based simulation. You have those opening a gym type thing or mujoco kind of simulated robots, and you have games, things like that. So that's where the field is going, really, this kind of environment. Now back to the question of AGI. I don't like the term AGI because it implies that human intelligence is general. And human intelligence is nothing like general. It's very, very specialized. We think it's general. We like to think of ourselves as having general intelligence. We don't, we're very specialized. We're only slightly more general than.

Why does it feel general? So you kind of the term general. I think what's impressive about humans is the ability to learn. As we were talking about learning to learn in just so many different domains. It's perhaps not arbitrarily general, but just you can learn in many domains and integrate that knowledge somehow. Okay, that knowledge persists.

So let me take a very specific example. Yes, it's not an example, it's more like a quasi mathematical demonstration. So you have about 1 million fibers coming out of one of your eyes. Okay, 2 million total. But let's talk about just one of them. It's 1 million nerve fibers, your optical nerve. Let's imagine that they are binary, so they can be active or inactive. So the input to your visual cortex is 1 million bits. Now they're connected to your brain in a particular way and your Brain has connections that are kind of a little bit like a convolutional net. They're kind of local in space and things like this. Now imagine I play a trick on you. It's a pretty nasty trick, I admit. I cut your optical nerve and I put a device that makes a random perturbation of a permutation of all the nerve fibers. So now what comes to your brain is a fixed but random permutation of all the pixels. There's no way in hell that your visual cortex, even if I do this to you in infancy, will actually learn vision to the same level of quality that you can.

Got it. And you're saying there's no way you've relearned that?

No, because now two pixels that are nearby in the world will end up in very different places in your visual cortex. And your neurons there have no connections with each other because they only connected locally.

So this whole, our entire. The hardware is built in many ways.

To support the locality of the real world.

Yeah.

Yes. That's specialization.

Yeah, but it's still pretty damn impressive. So it's not perfect generalization. It's not even close.

No, no, it's not that. It's not even close. It's not at all.

Yeah, it's not. It's specialized.

So how many Boolean functions? So let's imagine you want to train your visual system to recognize particular patterns of those 1 million bits. Okay, so that's a Boolean function, right? Either the pattern is here or not here. It's a two way classification with 1 million binary inputs. How many such Boolean functions are there? Okay, you have two to the 1 million combinations of inputs. For each of those you have an output bit. And so you have two to the 1 million Boolean functions of this type, which is an unimaginably large number. How many of those functions can actually be computed by your visual cortex? And the answer is a tiny, tiny, tiny, tiny, tiny, tiny sliver. Like an enormously tiny sliver.

Yeah.

So we are ridiculously specialized. Okay, but.

Okay, that's an argument against the word general. I think there's a. I agree with your intuition, but I'm not sure. It's. It seems the brain is impressively capable of adjusting to things.

It's because we can't imagine tasks that are outside of our comprehension. So we think we are general because we're general of all the things that we can apprehend. But there is a huge world out there of things that we have no idea. We call that heat by the Way heat. Heat. So at least physicists call that heat, or they call it entropy, which is, you know, you have a thing full of gas, right?

Closed system full of gas, right?

Closed or not closed, it has, you know, pressure, it has temperature, it has, you know, and you can write equations, PV equal NRT in things like that, right. When you reduce the volume, the temperature goes up, the pressure goes up, things like that. Right. For perfect gas, at least, those are the things you can know about that system. And it's a tiny, tiny number of bits compared to the complete information of the state of the entire system. Because the state of the entire system will give you the position and momentum of every molecule of the gas. And what you don't know about it is the entropy. And you interpret it as heat. The energy contained in that thing is what we call heat. Now it's very possible that in fact there is some very strong structure in how those molecules are moving. It's just that they are in a way that we are just not wired to perceive.

Yeah, we're ignorant to it. And there's near infinite amount of things we're not wired to perceive. And you're right, that's a nice way to put it. We're general to all the things we can imagine, which is a very tiny subset of all things that are possible.

So it's like Kolagov complexity or the combo of Chaitin sort of complexity. You know, every bit string or every integer is random except for all the ones that you can actually write down.

Yeah, okay. So beautifully put. But you know, so we can just call it artificial intelligence. We don't need to have a general.

Or human level, human level intelligence.

You know, you'll start anytime you touch human. It gets, it gets interesting because, you know, it's because we attach ourselves to human and it's difficult to define what human intelligence is.

Yeah.

Nevertheless, my definition is maybe dem impressive intelligence. Okay. Damn impressive demonstration of intelligence. Whatever.

Sa.