Moore's Law is Not Dead (Jim Keller) | AI Podcast Clips

For over 50 years now, Moore's Law has served for me and millions of others as an inspiring beacon of what kind of amazing future brilliant engineers can build. I'm just making your kids laugh all of today.

That's great.

So first, in your eyes, what is Moore's Law? If you could define, for people who don't know.

Well, the simple statement was from Gordon Moore was double the number of transistors every two years. Something like that. And then my operational model is we increase the performance of computers by 2x every two or three years. And it's wiggled around substantially over time, and also in how we deliver performance has changed. The foundational idea was 2x the transistors every two years. The current cadence is something like. They call it a shrink factor, like 0.6 every two years, which is not.

0.5, but that's referring strictly again to.

The original definition of transistor count.

And shrink factor is just getting them smaller, smaller, smaller.

Well, for a constant chip area, if you make the transistor smaller by 0.6, then you get one over 0.6 more transistors.

So can you linger on it a little longer? What's the. What's a broader. What do you think should be the broader definition of Moore's Law? When you mentioned how you think of performance just broadly, what's a good way to think about Moore's Law?

Well, first of all, so I. I've been aware of Moore's Law for 30 years.

In which sense?

Well, I've been designing computers for 40.

You're just watching it before your eyes kind of thing.

Well, and somewhere where I became aware of it, I was also informed that Moore's Law was going to die in 10 to 15 years. I thought that was true at first, but then after 10 years, it was going to die in 10 to 15 years. And then at one point, it was going to die in five years, and then it went back up to 10 years. And at some point I decided not to worry about that particular prognostication for the rest of my life, which is fun. And then I joined intel and everybody said, Moore's Law is dead. And I thought, that's sad because it's the Moore's Law company and it's not dead. And it's always been going to die. And humans like these apocryphal kind of statements like we'll run out of food or we'll run out of air or run out of room or run out of something.

Right. But it's still incredible that it's lived for as long as it has.

And.

And yes, there's many people who believe now that Moore's law is dead.

I know they can join the last 50 years of people who had the same.

Yeah, there's a long tradition. But why do you think, if you can try to understand it, why do you think it's not dead?

Well, first, let's just think. People think Moore's law is one thing. Transistors get smaller, but actually, under the sheet, there's literally thousands of innovations. And almost all those innovations have their own diminishing return curves. So if you graph it, it looks like a cascade of diminishing return curves. I don't know what to call that. But the result is an exponential curve. At least it has been. And we keep inventing new things. So if you're an expert in one of the things on a diminishing return curve and you can see its plateau, you will probably tell people, well, this is. This is done. Meanwhile, some other pilot people are doing something different. So that's just normal. So then there's the observation of how small could a switching device be? So a modern transistor is something like a thousand by one thousand by thousand atoms. Right. And you get quantum effects down around 2 to 10 atoms. So you can imagine a transistor as small as 10 by 10 by 10. So that's a million times smaller. And then the quantum computational people are working away at how to use quantum effects. So.

A thousand by thousand by thousand atoms. That's a really clean way of putting it.

Well, a fan, like a modern transistor, if you look at the fan, it's like 120 atoms wide, but we can make that thinner. And then there's. There's a gate wrapped around it, and then there's spacing. There's a whole bunch of geometry. And, you know, a competent transistor designer could count both atoms in every single direction. Like, there's techniques now to already put down atoms in a single atomic layer. Right. And you can place atoms if you want to. It's just, you know, from a manufacturing process, if placing an atom takes 10 minutes and you need to put 10 to the 23rd atoms together to make a computer, it would take a long time. So the methods are both shrinking things and then coming up with effective ways to control what's happening.

Manufacture stably and cheaply.

Yeah, so the innovation stack's pretty broad. There's. There's equipment, there's optics, there's chemistry, there's physics, there's material Science, there's metallurgy. There's lots of ideas about when you put different materials together, how do they interact, Are they stable? Are you stable over temperature, Are they repeatable? There's literally thousands of technologies involved.

But just for the shrinking, you don't think we're quite yet close to the fundamental limits of physics.

I did a talk on Moore's Law, and I asked for a roadmap to a path of 100. And after two weeks, they said, we only got to 50.

100. What, sorry?

100X shrink.

100X shrink. We only got to 50.

And I said, why don't you give it another two weeks? Well, here's the thing about Moore's Law, right. So I believe that the next 10 or 20 years of shrinking is going to happen. Right now, as a computer designer, you have two stances. You think it's going to shrink, in which case you're designing and thinking about architecture in a way that you'll use more transistors or conversely, not be swamped by the complexity of all the transistors you get. Right. You have to have a strategy so.

You'Re open to the possibility and waiting for the possibility of a whole new army of transistors ready to work.

I'm expecting more transistors every two or three years by a number large enough that how you think about design, how you think about architecture, has to change. Like, imagine you build buildings out of bricks, and every year the bricks are half the size, or every two years. Well, if you kept building bricks the same way, so many bricks per person per day, the amount of time to build a building would go up exponentially.

Right, Right.

But if you said, I know that's coming, so now I'm going to design equipment that moves bricks faster, uses them better, because maybe you're getting something out of the smaller bricks, more strength, thinner walls, less material efficiency out of that. So once you have a roadmap with what's going to happen. Transistors, we're going to get more of them, then you design all this collateral around it to take advantage of it and also to cope with it. Like, that's the thing people don't understand. It's like, if I didn't believe in Moore's Law and then Moore's Law transistors showed up, my design teams were all drowned.

So what's the hardest part of this inflood of new transistors? I mean, even if you just look historically, throughout your career, what's the thing? What fundamentally changes when you add more Transistors. In the task of designing an architecture.

Well, there's two constants, right? One is people don't get smarter.

By the way, there's some signs showing that we do get smarter because of nutrition, whatever. Sorry to bring that effect. Yes, yeah.

Familiar with it. Nobody understands it. Nobody knows if it's still going on. So that's a.

Or whether it's real or not.

But yeah, I sort of.

Anyway, but not explanation.

I would believe, for the most part, people aren't getting much smarter.

The evidence doesn't support it. That's right.

And then teams can't grow that much.

Right.

So human beings, we're really good in teams of 10, up to teams of 100, they can know each other. Beyond that, you have to have organizational boundaries. So those are pretty hard constraints. So then you have to divide and conquer. As the designs get bigger, you have to divide it into pieces. The power of abstraction layers is really high. We used to build computers out of transistors. Now we have a team that turns transistors into logic cells and another team that turns them into functional units, another one that turns them into computers. So we have abstraction layers in there. You have to think about, when do you shift gears on that? We also use faster computers to build faster computers. So some algorithms run twice as fast on new computers, but a lot of algorithms are N squared a computer with twice as many transistors and it might take four times as long to run. So you have to ref of the software. Like simply using faster computers to build bigger computers doesn't work.

So.

So you have to think about all these things.

So in terms of computing performance and the exciting possibility that more powerful computers bring is shrinking the thing we've just been talking about, one of the. For you, one of the biggest exciting possibilities of advancement in performance, or is there other directions that you're interested in? Like. Like in the direction of sort of enforcing given parallelism or like doing massive parallelism in terms of many, many CPUs, you know, stacking CPUs on top of each other. That kind of, that kind of parallelism or any kind of parallelism.

Well, think about it a different way. So old computers, you know, slow computers, you said a =b +c +d. Pretty simple, right? And then we made faster computers with vector units and you can do proper equations and matrices. Right. And then modern, like AI computations or like convolutional neural networks where you convolve one large data set against another. And so there's sort of this hierarchy of mathematics you know, from simple equation to linear equations to matrix equations to deeper kind of computation. And the data sets are getting so big that people are thinking of data as a topology problem. Data is organized in some immense shape and then the computation, which sort of wants to be get data from immense shape and do some computation on it. So what computers have allowed people to do is have algorithms go much, much further. So that paper you reference, the Sutton paper, they talked about, like when AI started, it was apply rule sets to something. That's a very simple computational situation. Then when they did first chess thing, they solved deep searches. So have a huge database of moves and results, deep search, but it's still just a search. Now we take large numbers of images and we use it to train these weight sets that we convolve across. It's a completely different kind of phenomena. We call that AI. Now they're doing the next generation, and if you look at it, they're going up this mathematical graph. Right. And then computations, both computation and data sets support going up that graph. Yeah.

The kind of computation though might. I mean, I would argue that all of it is still a search. Right? Just like you said, a topology problem of data sets. You're searching the data sets for valuable data. And also the actual optimization of neural networks is a kind of search for the.

I don't know, if you looked at the inner layers of finding a cat, it's not a search, it's a set of endless projections. So projection. Here's a shadow of this phone, right? And then you can have a shadow of that onto something. A shadow on that of something. If you look in the layers, you'll see this layer actually describes pointy ears and round eyeness and fuzziness. But the computation to tease out the attributes is not search. Right. I mean, like the inference part might be search, but the training's not search.

Okay, well, technically.

And then in deep networks, they look at layers and they don't even know it's represented. And yet if you take the layers out, it doesn't work. Okay, so I don't think it's search. All right, well, but you have to talk to a mathematician about what that actually is.

Oh, we could disagree, but the, the. It's just semantics. I think it's not, but it's certainly not.

I would say it's absolutely not semantics, but.

Okay, all right, well, if you want to go there. So optimization to me is search. And we're trying to optimize the ability of a neural network to detect cat ears and the Difference between chess and the space. The incredibly multidimensional hundred thousand dimensional space that neural networks are trying to optimize over is nothing like the chessboard database. So it's a totally different kind of thing. And okay, in that sense you can say it loses.

I can see how you might say the funny thing is, it's the difference between given search space and found search space.

Right, exactly.

Yeah, Maybe that's a different way to describe it.

That's a beautiful way to put it. Okay, but you're saying, what's your sense in terms of the basic mathematical operations and the architectures compute hardware that enables those operations? Do you see the CPUs of today still being a really core part of executing those mathematical operations?

Yes. Well, the operations continue to be add, subtract, load, store, compare and branch. It's remarkable. So it's interesting, the building blocks of computers or transistors under that atoms. So you got atoms, transistors, logic gates, computers, functional units of computers. The building blocks of mathematics at some level are things like adds and subtracts and multiplies. But the space mathematics can describe is, I think, essentially infinite. But the computers that run the algorithms are still doing the same things. Now, a given algorithm might say I need sparse data, or I need 32 bit data, or I need, you know, like a convolution operation that naturally takes eight bit data, multiplies it and sums it up a certain way. So the like the data types and tensorflow imply an optimization set. But when you go right down and look at the computers, it's an andor gates doing adds and multiplies like, like that hasn't changed much. Now the quantum researchers think they're going to change that radically. And then there's people who think about analog computing because you look in the brain and it seems to be more analog ish, you know, that maybe there's a way to do that more efficiently. But we have a million x on computation. And I don't know the relationship between computational, let's say intensity and ability to hit mathematical abstractions. I don't know anybody described that. But, but just like you saw in AI, you went from rule sets to simple search to complex search to say, found search, like those are, you know, orders of magnitude more computation to do. And as we get the next two orders of magnitude, like a friend, Roger Gadori said, like every order of magnitude.

Changes the computation fundamentally changes what the computation is doing.

Yeah. Oh, you know, the expression a difference in quantity is a difference in kind of, you know, the difference between ant and anthill. Right. Or neuron and brain. You know, there's, there's, there's, there's indefinable place where the, the quantity changed, the quality. Right. And we've seen that happen in mathematics multiple times. And, you know, my, my guess is it's going to keep happening.

So your sense is. Yeah, if you focus head down and shrinking the transistor.

Well, it's not just head down. We're aware of the software stacks that are running in the computational loads and we're kind of pondering what do you do with a petabyte of memory that wants to be accessed in a sparse way and have the kind of calculations AI programmers want. So there's a dialogue and interaction, but when you go in the computer chip, you find adders and subtractors and multipliers.

And so if you zoom out then with, as you mentioned, Ray Sutton, the idea that most of the development in the last many decades in AI research came from just leveraging computation and just simple algorithms waiting for the computation to improve.

Well, software guys have a thing that they call it the problem of early optimization. So you write a big software stack, and if you start optimizing, like the first thing you write, the odds of that being the performance limiter is low. But when you get the whole thing working, can you make it 2x faster by optimizing the right things? Sure. While you're optimizing that, could you have written a new software stack which would have been a better choice? Maybe now you have creative tension.

But the whole time as you're doing the writing, that's the software we're talking about, the hardware underneath gets faster.

Well, this goes back to the Moore's Law. If Moore's Law is going to continue, then your AI research should expect that to show up. And then you make a slightly different set of choices. Then we've hit the wall. Nothing's going to happen. And from here it's just us rewriting algorithms like that. Seems like a failed strategy for the last 30 years of Moore's Law's death.

So can you just linger on it? I think you've answered it, but I'll just ask the same dumb question over and over. So why do you think Moore's Law is not going to die? Which is the most promising, exciting possibility of why it won't die in the next five, 10 years? So is it the continuous shrinking of the transistor or is it another S curve that steps in and it totally.

Sort of shrinking the transistor? Is literally thousands of innovations. Right.

So there's S curves in there.

There's a whole bunch of S curves just kind of running their course and being reinvented and new things. You know, the, the semiconductor fabricators and technologists have all announced what's called nanowires. So they, they took a fan which had a gate around it and turned that into little wires so you have better control of that. And they're smaller. And then from there there's some obvious steps about how to shrink that. The metallurgy around wire stacks and stuff has very obvious abilities to shrink. And you know, there's a whole combination of things there to do.

Your sense is that we're going to get a lot if this innovation form just that shrinking.

Yeah, like a factor of 100 is a lot.

Yeah. I would say that's incredible. And it's totally, it's only 10 or 15 years now. You're smarter, you might know. But to me it's total unpredictable of what that 100x would bring in terms of the nature of the computation that people would be.

Yeah. Are you familiar with Bell's law? So for a long time it was mainframes, minis, workstation, PC, mobile. Moore's Law drove faster, smaller computers. Right. And then when we were thinking about Moore's Law, Raja Godori said every 10x generates a new computation. So scalar vector, matrix, topological computation. If you go look at the industry trends, there was mainframes and minicomputers and NPCs. And then the Internet took off and then we got mobile devices and now we're building 5G wireless with 1 millisecond latency. And people are starting to think about the smart world where everything knows you, recognizes you, the transformations are going to be unpredictable.

How does it make you feel that you're one of the key architects of this kind of future? So we're not talking about the architects of the high level people who build the Angry Bird apps and Angry Bird apps.

Who knows, maybe that's the whole point of the universe.

I'm going to take a stand at that. And the attention distracting nature of mobile phones. I'll take a stand. But anyway, in terms of that matters much? The side effects of smartphones or the attention distraction. Which part?

Well, who knows where this is all leading? It's changing so fast.

Wait, so back to My parents used.

To yell at my sisters for hiding in the closet with a wired phone with a dial on it. Stop talking to your friends all day. Right now my wife yells at my Kids for talking to their friends all day on text. It looks the same to me.

It's always, it's echoes of the same thing. Okay, but you are the one of the key people architecting the hardware of this future. How does that make you feel? Do you feel responsible? Do you feel excited?

So we're, we're in a social context. So there's billions of people on this planet. There are literally millions of people working on technology. I feel lucky to be, you know, doing what I do and getting paid for it. And there's an interest in it. But there's so many things going on in parallel. It's like the actions are so unpredictable. If I wasn't here, somebody else would do it. The vectors of all these different things are happening all the time. You know, there's a, I'm sure some philosopher or metaphilosophers, you know, wondering about how we transform our world.

So you can't deny the fact that these tools, whether these tools are changing our world.

That's right.

So do you think it's changing for the better?

Somebody. I read this thing recently. It said the, the two disciplines with the highest GRE scores in college are physics and philosophy. Right? And they're both sort of trying to answer the question, why is there anything right. And the philosophers, you know, are on the kind of theological side, and the physicists are obviously on the, you know, the material side. And there's a hundred billion galaxies with a hundred billion stars. It seems, well, repetitive at best. So I, you know, there's, there's on our way to 10 billion people. I mean, it's hard to say what it's all for, if that's what you're asking.

Yeah, I guess things do tend to.

Or significantly increases in complexity. And I'm curious about how computation, like our physical world, inherently generates mathematics. It's kind of obvious, right? So we have XYZ coordinates. You take a sphere, you make it bigger, you get a surface that falls, you know, grows by R squared, like it generally generates mathematics. And the mathematicians and the physicists have been having a lot of fun talking to each other for years. And computation has been, let's say, relatively pedestrian. Like computation in terms of mathematics has been doing binary algebra while those guys have been gallivanting through the other realms of possibility. Now, recently, the computation lets you do mathematical computations that are sophisticated enough that nobody understands how the answers came out.

Machine learning, Machine learning, it used to.

Be you get data set, you guess at a function, the function is considered physics if it's predictive of new functions, new data sets. Modern, you can take a large data set with no intuition about what it is and use machine learning to find a pattern that has no function. Right. And it can arrive at results that I don't know if they're completely mathematically describable. So computation has kind of done something interesting. Compared to A B C.