Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71

The following is a conversation with Vladimir Vapnik Part 2 the second time we spoke on the podcast. He's the co inventor of Support Vector Machines, Support Vector clustering, VC Theory and many foundational ideas in statistical learning. He was born in the Soviet Union, worked at the Institute of Control Sciences in Moscow, then in the US, worked at AT&T, NEC Labs, Facebook AI Research, and now is a professor at Columbia University. His work has been cited over 200,000 times. The first time we spoke on the podcast was just over a year ago, one of the early episodes. This time we spoke after a lecture he gave titled Complete Statistical Theory of Learning as part of the MIT series of lectures on deep learning and AI that I organized. I'll release the video of the lecture in the next few days. This podcast and lecture are independent from each other, so so you don't need one to understand the other. The lecture is quite technical and math heavy, so if you do watch both, I recommend listening to this podcast first since the podcast is probably a bit more accessible. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter lexfriedman spelled F R I D M A N. As usual, I'll do one or two minutes of ads now and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash app, the 1 finance app in the App Store. When you get it, use code lexpodcast Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Brokerage services are provided by Cash App Investing, a subsidiary of Square and member sipc. Since Cash App allows you to send and receive money digitally, peer to peer and security in all digital transactions is very important. Let me mention the PCI Data Security Standard PCI DSS Level 1 that Cash App is compliant with. I'm a big fan of standards for safety and security and PCI DSS is a good example of that where a bunch of competitors got together and agreed that there needs to be a global standard around the security of transactions. Now we just need to do the same for autonomous vehicles and AI systems in general. So again, if you get Cash App from the App Store or Google Play and use the code lexpodcast, you get $10 and cash app will also donate $10 to first, one of my favorite organizations that is helping to advance robotics and STEM education for young people around the world. And now here's my conversation with Vladimir Vapnik. You and I talked about Alan Turing yesterday a little bit. And that he, as the father of artificial intelligence, may have instilled in our field an ethic of engineering and not science, seeking more to build intelligence rather than to understand it. What do you think is the difference between these two paths of engineering intelligence and the science of intelligence?

It's completely different story. Engineering is imitation of human activity. You have to make a device which behave as human behave have all the functions of human. It does not matter how you do it. But to understand what is intelligence about is quite different problem. So I think, I believe that it's somehow related to predicate we talked yesterday about. Because look at the Vladimir Props idea. He just found 31. He predicates, he call it units which can explain human behavior, at least in Russian tales. You look at the Russian tales and derive from that and then people realize that it more wide than in Russian tales. It is in tv, in movie serials and so on and so on.

So you're talking about Vladimir propp, who in 1928 published a book, Morphology of the Folk Tale, describing 31 predicates that have this kind of sequential structure that a lot of the stories narratives follow in Russian folklore and in other context. We'll talk about it. I'd like to talk about predicates in a focused way. But let me, if you allow me to stay zoomed out on our friend Alan Turing. And you know, he inspired a generation with the. The Imitation Game.

Yes.

Do you think if we can linger in a little bit longer, do you think we can learn? Do you think learning to imitate intelligence can get us closer to the science to understanding intelligence? So why do you think imitation is so far from understanding?

I think that it is different between. You have different goals. So your goal is to create something, something useful. And that is great. And you can see how much things was done. And I believe that it will be done even more. It's self driving cars and also this business. It is great. And it was inspired by Turing vision. But understanding is very difficult. It more or less philosophical category. What means understand the world. I believe in him, which starts from Plato, that there exists vault of ideas. I believe that intelligence it is vault of ideas, but it is world of pure ideas. And when you combine them with reality things it creates, as in my case, invariance, which is very specific. And that's, I believe, the combination of ideas in way to constructing Invariant is intelligence. But first of all, predicate. If you know predicate and hopefully then not. Not too much predicate exists. For example, 31 predicate for human behavior. It is not a lot.

Vladimir Prop used 31, you can even call him predicates 31 predicates to describe stories, narratives. Do you think human behavior, how much of human behavior, how much of our world, our universe, all the things that matter in our existence can be summarized in predicates of the kind that Prop was working with?

I think that we have a lot of form of behavior, but I think that predicate is much less. Because even in this example which I gave you yesterday, you saw that predicate can be. One predicate can construct many different invariants depending on your data. They're applying to different data and they give different invariants. But pure ideas, maybe not so much, not so many. I don't know about that, but my guess, I hope that's why challenge about digital recognition, how much you need?

I think we'll talk about computer vision and 2D images a little bit in your challenge.

That's exactly about intelligence.

That's exactly. That's exactly about. No, that hopes to be exactly about the spirit of intelligence in the simplest possible way.

Absolutely. You should start the simplest way, otherwise you will not be able to do it.

Well, there's an open question whether starting at the mnist digit recognition is a step towards intelligence, or it's an entirely different thing.

I think that to beat records using say 100, 200 times less examples, you need intelligence.

You need intelligence. So let's. Because he used this term and it would be nice. I'd like to ask simple, maybe even dumb questions. Let's start with a predicate in terms of terms and how you think about it. What is a predicate?

I don't know. I have a feeling formally they exist. But I believe that predicate for 2D images, one of them is symmetry.

Hold on a second. Sorry. Sorry to interrupt and pull you back. At the simplest level, we're not being profound. Currently, a predicate is a statement of something that is true.

Yes.

Do you think of predicates as somehow probabilistic in nature? Or is this binary? This is truly constraints of logical statements about the world.

In my definition, the simplest predicate is function function. And you can use this function to make inner product that is predicated what's.

The input and what's the output of the function.

Input is x something which is input in reality. Say if you consider digit recognition, it pixel space input, but it Is function, which in pixel space. But it can be any function from pixel space. And you choose. And I believe that there are several functions which is important for understanding of images. One of them is symmetry. It's not so simple construction as I described with derivative, with all this stuff. But another I believe, I don't know how many is how. Well, structurized is picture.

Structurized?

Yeah.

What do you mean by structurized?

It is formal definition. Say something happened heavy on the left corner, not so heavy in the middle, and so on. You describe in general concept of what.

You assume concepts, some kind of universal concepts.

Yeah, but I don't know how to formalize this.

So this is the thing. There's a million ways we can talk about this. I'll keep bringing it up. But we humans have such concepts when we look at digits. But it's hard to put them just like you're saying now, it's hard to put them into words.

You know, that is example. When critics in music trying to describe music, they use predicate, and not too many predicate, but in different combination. But they have some special words for describing music. And the same should be for images. But maybe there are critics who understand essence of what this image is about.

Do you think there exists critics who can summarize the essence of images, human beings?

I hope so.

Yes, but explicitly state them on paper. The fundamental question I'm asking is, do you think there exists a small set of predicates that will summarize images? It feels to our mind like it does that the concept of what makes a two and a three and a four.

No, no, no. It's not on this level. It should not describe 2, 3, 4. It describes some construction which allow you.

To create invariance and invariance. Sorry to stick on this, but terminology.

Invariance, it is property of your image. I can say, looking on my image, it is more or less symmetric. And I can give you value of symmetry, say level of symmetry using this function which I gave yesterday. And you can describe that your image have these characteristics exactly in the way how musical critics describe music. So. But this is invariant applied to specific data, to specific music, to something. I strongly believe in this plot ideas that there exists world of predicate and world of reality. And predicate and reality is somehow connected. And you have to.

Let's talk about Plato a little bit. So you draw a line from Plato to Hegel to Wigner to today.

Yes.

So Plato has forms, the. The theory of forms. There's a world of Ideas and a world of things as you talk about. And there's a connection. And presumably the world of ideas is very small and the world of things is arbitrarily big. But they're all what Plato calls them like the. It's a shadow. The real world is a shadow from the world of.

Yeah, you have projection. Projection of world of idea.

Yeah, very poetic.

In reality, you can realize this projection using. Using this invariance because it is projection for. On specific examples which create specific features of specific objects. So.

And so the essence of intelligence is while only being able to observe the world of things, try to come up with a world of ideas exactly like.

In this music story. Intelligent musical critics knows this, all this work and have a feeling about what.

I feel like that's a contradiction. Intelligent music critics. But I think. I think music is to be enjoyed in all its forms. The notion of critic, like a food critic.

No, I don't want to touch emotion.

That's an interesting question. Does emotion. There's certain elements of the human psychology of the human experience which seem to almost contradict intelligence and reason. Like emotion, like fear, like. Like love. All of those things. Are those not connected in any way to the space of ideas.

Thus. I don't know. I just want to be concentrate on very simple story on digit recognition.

So you don't think you have to love and fear death in order to recognize digits?

I don't know because it's so complicated. It involved a lot of stuff which I never consider. But I know about digit recognition and I know that for digit recognition to get records from small number of observations, you need predicate, but not special predicate for this problem, but universal predicate which understand world of images, of visual information. Yeah, but on the first step they understand say world of handwritten digits or characters or something simple.

So like you said, symmetry is an interesting.

No, that's what I think. One of the predicates related to symmetry, but the level of symmetry.

Okay, Degree of symmetry. So you think symmetry at the bottom is a universal notion. And there's. There's degrees of a single kind of symmetry. Or is there many kinds of symmetries?

Many kinds of symmetries. There is a symmetry anti symmetry, say letter S. So it has vertical anti symmetry. And it could be diagonal symmetry. Vertical symmetry.

So when you cut vertically the letter S. Yeah.

Then the upper part and low part in different directions. Yeah.

Inverted along the Y axis. Yeah, but that's just like one example of symmetry. Right. Isn't there like.

Right. But there is a degree of symmetry. If you play all this lead derivative stuff to to do tangent distance, whatever I describe, you can have a degree of symmetry. And that is what describing reason of image. It is the same as you will describe this image. Saying about digits has Anti symmetry digits 3 Symmetric more or less look for symmetry.

Do you think such concepts like symmetry predicates like symmetry. Is it a hierarchical set of concepts? Or are these independent distinct predicates that we want to discover some set of?

No, there is a D of symmetry. And you can this idea of symmetry make very general like degree of symmetry. The degree of symmetry can be zero, no symmetry at all. Or degree of symmetry say more or less symmetrical. But in you have one of this description and symmetry can be different. As I told, horizontal, vertical, diagonal and anti Symmetry is also concept of symmetry.

What about shape in general? I mean, symmetry is a fascinating notion.

But no, no, I'm talking about digit. I would like to concentrate on all I would like to know predicate for digit recognition.

Yes, but symmetry is not enough for digit recognition, right?

It is not necessarily for digital recognition. It helps to create invariant which will. Which you can use when you will have examples for digit recognition. You have regular problem of digit recognition. You have examples of the first class or second class. Plus you know that there is this. There exists concept of symmetry and you apply. When you looking for decision rule. You will apply concept of symmetry of this level of symmetry which you estimate from me. So let's. Let's talk. Everything is comes from weak convergence.

What is convergence? What is weak convergence? What is strong convergence? I'm sorry, I'm going to do this to you. What are we converging from into.

You're converging. You would like to have a function the function which say indicator function which indicate your digit 5 for example, a classification task. Let's talk only about classification.

So classification means you will say whether this is a 5 or not or say which of the 10 digits it is.

Right? Right. I would like to have these functions. Then I have some examples. I can consider property of these examples. Say symmetry. And I can measure level of symmetry for every digit. And then I can take average and I from from my training data. And I will consider only functions of conditional probability which I'm looking for my decision rule which applying to digits will give me the same average as I absorb on training data. So actually this is different level of description of what you want. You want not just you show not one digit. You show this. This predicate show general property of all digits which you have in Mind. If you have in mind Digit 3, it gives you property of Digit 3 and you select as admissible set of function. Only function which keeps this property. You will not consider other functions. So you're immediately looking for smaller subset of function.

That's what you mean by admissible functions.

Admissible function. Exactly.

Which is still a pretty large for the number three.

It's pretty large. But if you have one predicate, but according to there is a strong and weak convergence. Strong convergence is convergence and function. You're looking at the function on one function and you're looking on another function. And square difference from them should be small. If you take difference in any points, make a square, make an integral, and it should be small. That is convergence in function. Suppose you have some function, any function. So I would say, I say that some function converge to this function if integral from square difference between them is small.

That's the definition of strong convergence. That definition of two functions. The integral of the difference.

Yes, small. It is convergence in functions. Yeah, but you have different convergence in functionals. You take any function, you take some function phi and take inner product, this function, this F function, F0 function which you want to find. And that gives you some value. So you say that set of functions converge in inner product to this function. If this value of inner product converge to value f0, that is for 1 Phi. But v convergence requires that it converges for any function of Hilbert space. If it converges for any function of Hilbert space, then you will say that this is weak convergence. You can think that when you take integral, that is integral property of function. For example, if you will take sine or cosine, it is coefficient of say Fourier expansion. So if it converge for all coefficients of Fourier expansion. So under some condition it converges to function you're looking for. But weak convergence means any property convergence. Not point wise, but integral property of function. So weak convergence means integral property of functions. When I talking about predicate, I would like to formulate which integral properties I would like to have for convergence. So and if I will take one predicated function which I measure property. If I will use one predicate and say I will consider only function which give me the same value as with this predicate. I selecting set of functions from functions which is admissible in the sense that function which I looking for in this set of functions because I checking in training data it gives the same.

Yes, it always has to be connected to the training data in terms of.

Yeah, but protecty you can know independent on training data. And this guy Prop. Yeah, so that there is formal property 31 property and fairy tale, Russian fairy tale. But Russian fairy tale is not so interesting. More interesting that people apply this to movies, to theater, to different things and the same works, they're universal.

So I would argue that there's a little bit of a difference between the kinds of things that were applied to which are essentially stories and digit recognition.

It is the same story.

You're saying digits. There's a story within the digit.

Yeah, but my point is why I hope that it's possible to beat a record using not 60,000, but but say 100 times less. Because instead you will give predicates and you will select your decision not from wide set of functions, but from set of function which keeps this predicate. But predicate is not related just to digit recognition.

Right?

Like in Plato's case.

Do you think it's possible to automatically discover the predicates? So you basically said that the essence of intelligence is the discovery of good predicates.

Yeah.

Now the natural question is, you know, that's what Einstein was good at doing in physics. Can we make machines do these kinds of discovery of good predicates or is this ultimately a human endeavor?

Thus I don't know. I don't think that machine can do. Because according to theory about weak convergence, any function from Hilbert space can be predicated. So you have infinite number of predicate in upper and before you don't know which predicate is good on which. But whatever Prop. Show and why people call it breakthrough that there is not too many predicate which cover most of situation happened in the world.

So there's a sea of predicates and most of the only a small amount are useful for the kinds of things that that happen in the world.

I think that I would say only a small part of predicate. Very useful. Useful all of all of them.

Only very few are what we should let's call them good predicates.

Very good predicates.

Very good predicates. So can we linger on it? What's your intuition? Why is it hard for a machine to discover good predicates?

I even in my talk described how to do predicate have to find new predicate. I'm not sure that it is very good.

What did you propose in your talk?

No, in my talk I gave example for diabetes.

Diabetes.

When we achieve some percent. So then we're looking for area where some sort of predicate which I formulate does not keeps invariant. So if it doesn't keep, I I retrain my data. I select Only function which keeps this invariant in the way I did it, I improve my performance. I can looking for this predicate. I know technically how to do that. And you can of course do it using machine. But I'm not sure that we will construct the smartest predicate.

Well, this is the. Allow me to linger on it because that's the essence. That's the challenge. That is artificial. That's the human level intelligence that we seek is the discovery of these good predicates. You've talked about deep learning as a way to the predicates they use. And the functions are mediocre. We can find better ones.

Let's talk about deep learning.

Sure, let's do it.

I know only Jan slicun convolutional network. And what else I don't know. And it's a very simple convolution. There's not much else to know left and right. Yes, I can do it like that. This one predicate.

It is convolution is a single predicate.

It's single. It's. It's single predicate. Yes, you know, exactly. You take lead derivative for translation and predicate. This should.

Should be kept so that's a single predicate. But humans discovered that one or at.

Least not that is a risk not too many predicates. And that is big story because Jan did it 25 years ago and nothing so clear was added to to deep network. And then I don't understand why we should talk about deep network instead of talking about piecewise linear functions which keeps this predicate.

Well, the. You know, a counter argument is that maybe the amount of predicates necessary to solve general intelligence, say in the space of images doing efficient recognition of handwritten digits is very small. And so we shouldn't be so obsessed about finding. We'll find other good predicates like convolution, for example. You know, there has been other advancements like if you look at the work with attention there's a tensional mechanisms in especially used in natural language focusing the network's ability to. To learn at which part of the input to look at. The thing is, there's other things besides predicates that are important for the actual engineering mechanism of showing how much you can really do given such these predicates. It. I mean that's essentially the work of deep learning is constructing architectures that are able to be given the training data to be able to converge towards a function that can approximate. Can generalize.

Well, it's an engineering problem. Yeah, I understand. But let's talk not on emotional level, but on a mathematical level. You have set of piecewise linear functions. It is all possible neural networks. It's just piecewise linear functions. There's many, many pieces large, large number of piecewise. Okay, exactly.

But very large.

Very large almost. But it feels like too large. It's still simpler than say convolution than reproducing kernel Hilbert space, which have a Hilbert set of functions.

What's Hilbert space?

It's space with infinite number of coordinates function for expansion, something like that. So it's much richer. So. And when I talking about closed form solution, I talking about this set of function not piecewise linear set, which is particular case. It is small part.

The neural networks is a small part of the space. You're talking of functions.

You're talking about say small set of functions. Let me take it. But it is fine. It is fine. I don't want to discuss the small or big take advantage. So you have some set of functions. So now when you're trying to create architecture, you would like to create admissible set of functions. All your tricks to use not all functions, but some subset of this set of functions. Say when you're introducing convolutional net, it is way to make this subset useful for you. But from my point of view, convolutional it is something you want to keep some invariants, say translation invariance. But now if you understand this and you cannot explain on the level of ideas what neural network does, you should agree that it is much better to have a set of functions. And they say this set of functions should be admissible. It must keep this invariant, this invariant and that invariant. You know that as soon as you incorporate new invariance set of function. Because smaller and smaller and smaller.

But all the invariants are specified by you, the human.

Yeah, but what I am hope that there is a standard predicate like prop show that what that's what I want to find for digital recognition. If we start it is completely new area. What is intelligence about on the level starting from. From Plata's idea. What is world of ideas? So. And I believe that it's not too many. Yeah, but you know, it is amusing that mathematician doing something in neural network in. In general function. But people from literature, from art, they use this all the time.

That's right.

In variance saying say it is great. Have. Have people describe music. We should learn from that in something on this level. But so why Vladimir Prop, who was just theoretical, who studied theoretical literature, he.

Found that you know what? Let Me throw that right back at you because there's a little bit of a that's less mathematical and more emotional philosophical Vladimir prop. I mean, he wasn't doing math.

No.

And you just said another emotional statement, which is you believe that this Plato world of ideas is small.

I hope.

I hope.

Do.

No, do you do. What's your intuition though? If it can linger on it, you.

Know, because it's not just small or big. I know, exactly. Then when I introducing some predicate, I decrease set of functions. But my goal to decrease set of.

Function much by as much as possible.

By as much as possible good predicate which does this, then I should choose next predicate which does. Which decreases it as much as possible. So set of good predicates, it is such that they decrease this amount of admissible functions.

So if each good predicate significantly reduces the set of admissible functions that there naturally should not be that many good predicates.

No, but if you reduce very well the VC dimension of the function of admissible set of function is small and you need not too much training data to do well.

And VC dimension by the way, is some measure of capacity of this set of functions, right?

Roughly speaking, how many function in this set? So you're decreasing, decreasing and it makes easy for you to find function you're looking for. But the most important part to create good admissible set of functions. And it probably there are many ways but the. The good predicated such that. That can do that. So for. For. For this duck, you should know a little bit about duck, because what are the.

What are the three fundamental laws of ducks?

Looks like a duck, swims like a duck.

And you should know something about ducks.

To be able to not necessarily looks like say horse. It's also good.

It generalizes yes, from ducks and talk.

Like and make sound like horse something and run like horse and moves like horse. It is general. It is general predicate that this applied to doc. But for Doc, you can say play chess like duck.

You cannot say play chess.

Why not?

So you're saying you can, but that would not be a good.

No, you will not reduce a lot of functions.

You would not do. Yeah, you would not reduce the set of functions.

So the story is formal story, mathematical story is that you can use any function you want as a predicate. But some of them are good, some of them are not, because some of them reduce a lot of functions to admissible set of some of them.

So but the question is, and I'll probably keep asking this question, but how do we Find such. What's your intuition? Handwritten. Handwritten recognition. How do we find the answer to your challenge?

Yeah, yeah, I understand it like that. I understand vogue. What.

What defined.

What it means. I. I knew predicate.

Yeah.

Like guy who understand music can say this word which he described him when he listened to music. He understand music. He use not too many different. Oh, you can do like prop. You can make collection. What he talking about music about this, about that. It's not too many different situations he described.

Because we mentioned Vladimir prop a bunch. Let me just mention there's a. So there's a sequence of 31 structural notions that are common in stories.

And I think you call it units.

Units. And I think they resonate. I mean, it starts just to give an example. Absention. A member of the hero's community, a family leaves the security of the home environment. Then it goes to the interdiction. A forbidding edict or command is passed upon the hero. Don't go there. Don't do this. The heroes warned against some action. Then step three. Violate violation of interdiction. Break the rules, break out on your own. Then reconnaissance. The villain makes an effort to attain knowledge, needing to fulfill their plot. So on it goes on like this. Ends. Ends in a wedding. Number 31.

Yeah.

Happily ever after.

No, he. He just gave description of all situation. He understands this vault of folk tales. Yeah, not folk, but it folks stories. And this story is not in just folk tales. This story is in. In detective serials as well.

And probably in our lives. We probably live.

Read this at the end. They wrote that this predicate is good for different situation from movie for movie, for theater.

By the way, there's also criticism, right? There's an other way to interpret narratives from Claude Levy Strauss.

I'm not in this business.

No, I know it's theoretical literature, but it's looking at paradise.

It's always the discussion. Yeah, but at least there is a units. It's not too many units that can describe, but this guy probably gives another units or another way. Of course.

Exactly. Another set of units, another set of predictions.

It does not matter. But they exist. Probably.

My. My question is whether given those units, whether without our human brains to interpret these units, they would still hold as much power as they have meaning. Are those units enough when we give them to the alien species?

Let me ask you. Do you understand digit recognition? Digitized by Google Images?

No, I don't understand.

No, no, no. When you can recognize these digit images, it means that you understand.

Yes.

You understand characters, you understand.

No, no, no. No, I, I, it's, it's the imitation versus understanding question. Because I don't understand the mechanism by which.

No, I'm talking about predicates. You understand that it involves symmetry, maybe structure, maybe something else I cannot formulate. I just was able to find symmetries.

That's really good. So this is a good line. I feel like I understand the basic elements of what makes a good hand recognition system. My own like symmetry connects with me. It seems like that's a very powerful predicate. My question is, is there a lot more going on that we're not able to introspect? Maybe I need to be able to understand a huge amount in the world of ideas, thousands of predicates, millions of predicates in order to do hand recognition.

I don't think so.

So you're, you're both your hope and your intuition are such that.

Let me explain.

Enough.

You're using digits, you're using examples as well. Theory says that if you will use all possible functions from Hilbert space, all possible predicate, you don't need training data, you just will have admissible set of functions which contain one function.

Yes. So the trade off is when you're not using all predicates, you're only using a few good predicates. You need to have some training data.

Yes, exactly.

The more good predicates you have, the less training data.

Exactly. That is intelligent learning still.

Okay, I'm going to keep asking the same dumb question. Handwritten recognition to solve the challenge. You kind of propose a challenge that says we should be able to get state of the art MNIST error rates by using very few 60, maybe fewer examples per digit. What kind of predicates do you think you'll solve?

That is the challenge. So people who will solve this problem, they will answer. They will answer.

Do you think they'll be able to answer it in a human explainable way?

They just need to write function. That's it.

But so can that function be written, I guess by an automated reasoning system? Whether we're talking about a neural network learning a particular function or another mechanism.

No, I'm not against neural network. I am against admissible set of function which create neural network. You did it by hand. You don't do it by invariance, by predicate, by reason.

But neural networks can then reverse do the reverse step of helping you find a function. The task of a neural network is to find a disentangled representation. For example they call is to find that one predicate function that's really captures some kind of essence one, not the entire essence, but one very useful essence of this particular visual space. Do you think that's possible? Like listen, I'm. I'm grasping, hoping there's an automated way to find good predicates, right? So the question is, what are the mechanisms of finding good predicates Ideas you think we should pursue A young grad student listening right now?

I gave example. So find situation where predicate which you suggesting don't create invariant. It's like in physics, find situation where existing theory cannot explain it.

Find situation where the existing theory cannot explain this. So you're finding contradictions.

Find contradiction and then remove this contradiction. But in my case, what means contradiction? You find function which if you will use this function, you're not keeping invariants.

So really the process of discovering contradictions.

Yeah, it is like in physics. Find situation where you have contradiction for one of the property for one of the predicate. Then include this predicate making invariance and solve again this problem. Now you don't have contradiction, but it is not the best way. Probably. I don't know to looking for predicate.

That's just one way.

Okay, that. No, no, it is brute force way.

The brute force way. What about the ideas of some what big umbrella term of symbolic AI there's what in the 80s with expert systems, sort of logic reasoning based systems. Is there hope there to find some through sort of deductive reasoning to find good predicates?

I don't think so. I think that just logic is not enough.

It's kind of a compelling notion though, you know that when smart people sit in a room and reason through things, it seems compelling. And making our machines do the same is also compelling.

So everything is very simple. When you have infinite number of predicate, you can choose the function you want. You have invariance and you can choose the function you want. But you have to have not too many invariants to solve the problem. So in half from infinite number of function to select finite number and hopefully small number of functions which is good enough to extract small set of admissible functions. So they will be admissible. It's for sure. Because every function just decrease set of function and leaving it admissible. But it will be small.

But why do you think logic based systems don't. Can't help intuition.

Not because you should know reality, you should know life. This guy like Prop, he knows something and he tried to to put in invariant his understanding.

So. But that's the human. Yeah, but see you're. You're putting too much value into Vladimir Props. Knowing something.

No. It is what means, you know, life.

What it means, you know, common sense.

No, no. You know something common sense. It is some rules.

You think so? Common sense is simply rules. Common sense is every. It's mortality, it's no, it's, it's fear of death, it's love, it's spirituality, it's happiness and sadness. All of it is tied up into understanding gravity, which is what we think of as common sense.

I don't read it to discuss so white. I want to discuss understand digital. Understand digital recognition.

Anytime I bring up love and death, you bring it back to digital recognition.

No. You know, it is durable. Because there is a challenge.

Yeah.

Which I see how to solve it. If I will have a student concentrate on this work, I will suggest something to solve.

You mean handwritten recognition? Yeah. It's a beautifully simple, elegant and yet.

I think that I know invariance which will solve this.

You do?

I think so. You think so. But it is not universal. It is maybe I want some universal invariants which are good not only for digit recognition, for image understanding.

So let me ask, how hard do you think is 2D image understanding? So if we, we can kind of intuit handwritten recognition, how big of a step, leap journey is it from that? If I gave you good, if I solved your challenge from handwriting recognition, how long would my journey then be from that to understanding more general natural images?

Immediately. You will understand this as soon as you will make a record because it is not for free. As soon as you will create several invariants which will help you to get the same performance that the best neural net did using hundred and maybe more than 100 times less examples. You have to have something smart to do that.

And you're saying that that is invariant.

It is predicated because you should put some idea how to do that.

But okay, let me just pause. Maybe it's a trivial point, maybe not. But handwritten recognition feels like a 2D two dimensional problem. And it seems like how much complicated is the fact that most images are projection of a three dimensional world onto a 2D plane. It feels like for a three dimensional world we need to start understanding common sense in order to understand an image. It's no longer visual shape and symmetry. It's having to start to understand concepts of. Understand life.

Yeah. You're talking that there are different invariant, different predicates. Yeah.

And potentially much larger number.

You know, might be. But let's start from simple.

Yeah, but you said that, you know.

I, I, I cannot think.

Yes.

About things Which I don't understand. This I understand, but I'm sure that I don't understand everything there. Yeah, that's a different stool. As simple as possible, but not simpler. And that is exact case with handwritten.

With handwritten, yeah, but never. That's the difference between you and I. I, I, I welcome and enjoy thinking about things I completely don't understand because to me it's a natural extension, without having solved handwritten recognition, to wonder how difficult is the next step of understanding 2D 3D images? Because ultimately, while the science of intelligence is fascinating, it's also fascinating to see how that maps to the engineering of intelligence. And recognizing handwritten digits is not. Doesn't help you. It might, it may not help you with the problem of general intelligence. We don't know. It'll help you a little bit.

We don't know.

It's unclear. Yeah, yeah, it might very.

But I would like to make a remark. Yes. I start not from very primitive problem. Make a challenge problem. I start with very general problem with Plato. So you understand, and it comes from Plato to digit recognition.

So you basically took Plato and the world of forms and ideas and mapped and projected into the clearest, simplest formulation of that big world.

I would say that I did not understand Plato until recently and until I consider the.