Jim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162
Jim Keller discusses how branch prediction and cache design hit physical limits, why craftsmanship matters in chip design, and his new work on graph-native AI hardware. He traces how Intel and ARM shaped computing, and why Moore's Law now means scaling across many machines instead of faster transistors.
Chapters
The following is a conversation with Jim Keller, his second time in the podcast. Jim is a legendary microprocessor architect and is widely seen as one of the greatest engineering minds of the computing age. In a peculiar twist of space time in our simulation, Jim is also a brother in law of Jordan Peterson. We talk about this and about computing, artificial intelligence, consciousness and life. Quick mention of our sponsors Athletic greens all in one nutrition drink, Brooklyn and Sheets ExpressVPN and Bel Campo Grass Fed Meat. Click the sponsor links to get a discount and to support this podcast. As a side note, let me say that Jim is someone who on a personal level, inspired me to be myself. There was something in his words on and off the mic, or perhaps that he even paid attention to me at all, that almost told me, you're alright kid, A kind of pat on the back that can make the difference between a mind that flourishes and a mind that is broken down by the cynicism of the world. So I guess that's just my brief few words of thank you to Jim and in general gratitude for the people who have given me a chance on this podcast, in my work and in life. If you enjoy this thing, subscribe on YouTube, review it on Apple Podcasts, follow on Spotify, support it on Patreon, or connect with me on Twitter lexfriedman. And now here's my conversation with Jim Keller.
What's the value and effectiveness of theory versus engineering? This dichotomy in building good software or hardware systems?
Well, it's good designs. Both, I guess that's pretty obvious. By engineering do you mean reduction of practice of known methods? And then science is the pursuit of discovering things that people don't understand or solving unknown problems.
Definitions are interesting here, but I was thinking more in theory, constructing models that kind of generalize about how things work. Engineering is actually building stuff. The pragmatic like, okay, we have these nice models, but how do we actually get things to work? Maybe economics is a nice example. Like economists have all these models of how the economy works and how different policies will have an effect. But then there's the actual, let's call it engineering of actually deploying the policies.
So computer design is almost all engineering and reduction of practice of known methods. Now, because of the complexity of the computers we built, you could think you're, well, we'll just go write some code and then we'll verify it and then we'll put it together and then you find out that the combination of all that stuff is complicated and then you have to be inventive to figure out how to do it. So that's definitely happens a lot. And then every so often some big idea happens, but it might be one person.
And that idea is in the space of engineering, or is it in the space of.
Well, I'll give you an example. So one of the limits of computer performance is branch prediction. So. And there's a whole bunch of ideas about how good you could predict the branch. And people said there's a limit to it, it's an asymptotic curve. And somebody came up with a better way to do branch prediction that was a lot better. And he published a paper on it. And every computer in the world now uses it. And it was one idea. So the engineers who build branch prediction hardware were happy to drop the one kind of training array and put it in another one. So it was a real idea.
And branch prediction is one of the key problems underlying all of sort of the lowest level of software.
It boils down to branch prediction, boils down to uncertainty. Computers are limited by, you know, single thread. Computers are limited by two things, the predictability of the path of the branches and the predictability of the locality of data. So we have predictors that now predict both of those pretty well. Yeah. So memory is, you know, a couple hundred cycles away. Local cache is a couple cycles away. When you're executing fast, virtually all the data has to be in the local cache. So a simple program says, you know, add one to every element in an array. It's really easy to see what the stream of data will be. But you might have a more complicated program that, you know, says, get a, get an element of this array, look at something, make a decision, go get another element. It's kind of random and you could think that's really unpredictable. And then you make this big predictor that looks at this kind of pattern and you realize, well, if you get this data and this data, then you probably want that one. And if you get this one and this one and this one, you probably want that one.
And is that theory or is that engineering like the paper that was written? Was it asymptotic kind of discussion, or is it more like here's a hack.
That works well, it's a little bit of both. Like there's information theory in it, I think, somewhere.
Okay, so it's actually trying to prove.
Yeah, but once, once, you know, the method implementing it is an engineering problem. Now there's a flip side of this, which is in a big design team, what percentage of people think their, their, their, their, their Their plan or their life's work is engineering versus design, inventing things. So lots of companies will reward you for filing patents? Yes, Some. Many big companies get stuck because to get promoted you have to come up with something new. And then what happens is everybody's trying to do some random new thing, 99% of which doesn't matter. And the basics get neglected or they get to. There's a dichotomy. They think like the cell library and the basic CAD tools or basic software validation methods. That's simple stuff. They want to work on the exciting stuff, and then they spend lots of time trying to figure out how to patent something. And that's mostly useless.
But the breakthroughs are on the simple stuff.
No, no, you have to do the simple stuff really well. If you're building a building out of bricks, you want great bricks. So you go to two places that sell bricks, and one guy says, yeah, they're over there in an ugly pile. And the other guy lovingly tells you about the 50 kinds of bricks and how hard they are and how beautiful they are and how square they. And you know, which one are you going to buy bricks from which is going to make a better house?
So you're talking about the craftsman, the person who understands bricks, who loves bricks, who loves the variety.
That's a good word. You know, good engineering is great craftsmanship. And when you start thinking engineering is about invention and you set up a system that rewards invention, the craftsmanship gets neglected.
Okay, so maybe one perspective is the theory. The science over emphasizes invention and engineering emphasizes craftsmanship. And therefore, like, so if you. It doesn't matter what you do theory.
Well, everybody does, like read the tech rags. They're always talking about some breakthrough or innovation, and everybody thinks that's the most important thing. But the number of innovative ideas is actually relatively low. But we need them, right? And innovation creates a whole new opportunity. Like when some guy invented the Internet, right? Like that was a big thing. The million people that wrote software against that were mostly doing engineering software writing. So the elaboration of that idea was huge.
I don't know if you know Brendan Eich, he wrote JavaScript in 10 days. And that's an interesting story. It makes me wonder. And it was famously for many years considered to be a pretty crappy programming language. Still is, perhaps it's been improving sort of consistently. But the interesting thing about that guy is, you know, he doesn't get any awards. You don't get a Nobel Prize or a fields medal or 19, a crappy.
Piece of you know, that software code.
That, that is currently the number one programming language in the world and runs now, is increasingly running the back end of the Internet.
Does he know why everybody uses it? That would be an interesting thing. Was it the right thing at the right time? Because when stuff like JavaScript came out, there was a move from writing C programs in C to what they call managed code frameworks, where you write simple code, it might be interpreted, it has lots of libraries, productivity is high and you don't have to be an expert. So Java was supposed to solve all the world's problems. It was complicated. JavaScript came out after a bunch of other scripting languages. I'm not an expert on it, but. Yeah, but was it the right thing at the right time or was there something, you know, clever? Because he wasn't the only one.
There's a few elements.
Maybe if he figured out what it was, then he'd get a prize like that.
Constructive theory.
Yeah. You know, maybe his problem is he hasn't defined this or he just needs a good promoter.
Well, I think there was a bunch of blog posts written about it, which is like wrong is right, which is like doing the crappy thing fast, just like hacking together the thing that answers some of the needs and then iterating over time. Listening to developers, like listening to people who actually use the thing. This is something you can do more in software, but the right time, like you have to sense, you have to have a good instinct of when is the right time for the right tool and make it super simple and just get it out there. The problem is this is true with hardware, this is less true with software, is there's backward compatibility that just drags behind you as you know, as you try to fix all the mistakes of the past. But the timing was good. There's something about that and it wasn't accidental. You have to give yourself over to the. You have to have this broad sense of what's needed now, both scientifically and like the community. And just like this, it was obvious that there was no. The interesting thing about JavaScript is everything that ran in the browser at the time, like Java and I think other, like Scheme, other programming languages, they were all in a separate external container. And then JavaScript was literally just injected into the web page. It was the dumbest possible thing running in the same thread as everything else. And like it was inserted as a comment. So JavaScript code is inserted as a comment in the HTML code. And it was, I mean, it's, there's, it's either Genius or super dumb. But it's like, right, so it has.
No apparatus for like a virtual machine and container. It just executed in the framework, the program is already running and it was, that's cool.
And then because something about that accessibility, the ease of its use, resulted in then developers innovating of how to actually use it. I mean, I don't even know what to make of that, but it does seem to echo across different software, like stories of different software. PHP has the same story, really crappy language, they just took over the world.
Always have a joke that the random length instruction set, variable length instruction sets, always one, even though they're obviously worse. Like nobody knows why. X86 is arguably the worst architecture on the planet. This is one of the most popular ones.
Well, I mean, isn't that also the story of risk versus I mean, is that simplicity? There's something about simplicity that us in this evolutionary process is valued. If it's simple, it spreads faster, it seems like. Or is that not always true?
That's not always true. Yeah, it could be. Simple is good, but too simple is bad.
So why did Risk win, you think.
So far, did risk win in the.
Long arc of history?
Maybe. We don't know.
So who's going to win? What's risk? What's cisc? And who's going to win in that space? In these instruction sets?
AI software is going to win, but there'll be little computers that run little programs like Normal all over the place. But, but we're, we're going through another transformation, so.
But you think instruction sets underneath it all will change?
Yeah, they evolve slowly. They don't matter very much.
They don't matter very much. Okay.
I mean the limits of performance are, you know, predictability of instructions and data. I mean that's the big thing. And then the usability of it is some, you know, quality of design, quality of tools, availability. Like right now, x86 is proprietary with intel and AMD, but they can change it any way they want independently. ARM is proprietary to arm and they won't let anybody else change it. So it's like a sole point. And RISC V is open source, so anybody can change it, which is super cool. But that also might mean it gets changed in too many random ways, that there's no common subset of it that people can use.
Do you like open or do you like closed? Like if you were to bet all your money on one or the other risk five versus it?
No idea.
It's case dependent.
Well, x86, oddly enough, when intel first started developing it, they licensed it like seven people. So it was the open architecture. And then they moved faster than others and also bought one or two of them. But there was seven different people making X86 because at the time there was 6502 and Z80s and you know, 8086 and you could argue everybody thought Z80 was the better instruction set. But that was proprietary to one place. Oh, in the 6800. So there's like five different, four or five different microprocessors. Intel went open, got the market share because people felt like they had multiple sources from it. And then over time it narrowed down to two players.
So why you as a historian, why did intel win for so long with their processors?
I mean they were great, their process development was great.
So it's just looking back to JavaScript and Brendan is Microsoft and Netscape and all these Internet browsers. Microsoft won the browser game because they aggressively stole other people's ideas. Like right after they did it, you.
Know, I don't know if intel was stealing other people's ideas.
They started in a good way stealing.
In a good way. Just to clarify, they started making rams random access memories. And then at the time when the Japanese manufacturers came up, you know, they were getting out, competed on that and they pivoted the microprocessors and they made the first, you know, integrated microprocessor programs. It was the 4004 or something.
Who was behind that pivot? That's a hell of a pivot.
Andy Grove and he was great.
That's a hell of a pivot.
And then they led semiconductor industry. Like they were just a little company. IBM, all kinds of big companies had boatloads of money and they out innovated.
Everybody auto innovated.
Okay, yeah, yeah.
So it's not like marketing. It's not.
Their processor designs were pretty good. I think the, you know, Core 2 was probably the first one I thought was great. It was a really fast processor. And then Haswell was great.
What makes a great processor in that?
Oh, if you just look at its performance versus everybody else, it's, you know, the size of it, the, you know, usability of it.
So it's not specific, some kind of element that makes it beautiful. It's just like literally just raw performance. Is that how you think of bioprocessors? It's just like raw performance?
Of course. It's like a horse race. The fastest one wins.
Now you don't care how as long as it wins.
Well, there's the Fastest in the environment. Like for years you made the fastest one you could and then people started to have power limits, so then you made the fastest at the right power point. And then when we started doing multi processors, if you could scale your processors more than the other guy, you could be 10% faster on like a single thread, but you have more threads, so there's lots of variability. And then ARM really explored like, you know, they have the A series and the R series and the M series, like a family of processors for all these different design points from like unbelievably small and simple. And so then when you're doing the design, it's sort of like this big pallet of CPUs, like they're the only ones with a credible, you know, top to bottom palette.
And what do you mean a credible top to bottom?
Well, there's people who make microcontrollers that are small but they don't have a fast one. There's people make fast processors but don't have a little, a medium one or a small one.
Is that hard to do? That full palette, that's.
That seems like a. Yeah, it's a lot of different.
So what's the difference in the ARM folks and intel in terms of the way they're approaching this problem?
Well, intel, almost all their processor designs were very custom, high end for the last 15, 20 years.
The fastest horse possible in one horse race.
Yeah, and architecturally they're really good, but the company itself was fairly insular to what's going on in the industry with CAD tools and stuff. And there's this debate about custom design versus synthesis. And how do you approach that? I'd say intel was slow on the getting to synthesize processors. ARM came in from the bottom and they generated IP which went to all kinds of customers. So they had very little say in how the customer implemented their ip. So ARM is super friendly to the synthesis IP environment, whereas intel said we're going to make this great client chip, server chip with our own CAD tools, with our own process, with our own other supporting ip and everything only works with our stuff.
So is ARM winning the mobile platform.
Space in terms of process?
And so in that what you're describing is why they're winning?
Well, they had lots of people doing lots of different experiments, so they controlled the processor architecture and ip, but they let people put in lots of different chips and there was a lot of variability in what happened there. Whereas intel, when they made their mobile, their foray into mobile, they had one team doing one part Right. So it wasn't 10 experiments. And then their mindset was PC mindset, Microsoft software mindset. And that brought a whole bunch of things along that the mobile world, the embedded world don't do.
Do you think it was possible for intel to pivot hard and win the mobile market? That's a hell of a difficult thing to do. Right, For a huge company to just pivot. I mean, so interesting to. Because we'll talk about your current work. It's like, it's clear that PCs were dominating for several decades, like desktop computers and then mobile. It's unclear.
It's a leadership question. Like Apple under Steve Jobs, when he came back, they pivoted multiple times. They built iPads and itunes and phones and tablets and great Macs. Like, who knew computers should be made out of aluminum? Nobody knew that. That they're great. It's super fun.
That was Steve.
Yeah, Steve Jobs. Like, they pivoted multiple times. And you know, the old intel, they. They did that multiple times. They made DRAMs and processors and processes.
And I gotta ask this. What was it like working with Steve Jobs?
I didn't work with him.
Did you interact with him?
Twice? I said hi to him twice in the cafeteria.
What did he say hi?
He said, hey, fellas. He was friendly. He was wandering around with somebody. He couldn't find a table because the cafeteria was packed. And I gave him my table. But I worked for Mike Colbert, who talked to. Mike was the unofficial CTO of Apple and a brilliant guy. And he worked for Steve for 25 years, maybe more. And he talked to Steve multiple times a day. And he was one of the people that could put up with Steve's, let's say, brilliance and intensity. And Steve really liked him. And Steve trusted Mike to translate the shit he thought up into engineering products that work. And then Mike ran a group called Platform Architecture, and I was in that group so many times I'd be sitting with Mike and the phone would ring, it'd be Steve. And Mike would hold the phone like this because Steve would be yelling about.
Something or other, and he would translate.
And he'd translate and then he would say, steve wants us to do this.
So was Steve a good engineer or. No?
I don't know. He was a great idea guy, idea person, and he's a really good selector for talent.
Yeah, that seems to be one of the key elements of leadership.
Right. And then he was a really good first principles guy. Like. Like somebody say something couldn't be done, and he would just think, that's obviously wrong. Right. But, you know, maybe it's hard to do, maybe it's expensive to do, maybe we need different people. There's a whole bunch of. If you want to do something hard, maybe it takes time, maybe you have to iterate. There's a whole bunch of things you could think about, but saying it can't be done is stupid.
How would you compare? So it seems like Elon Musk is more engineering centric, but I think he considered himself a designer too. He has a design mind. Steve Jobs feels like he is much more idea space, design. Space versus engineering. Just make it happen. Like the world should be this way. Just figure it out.
But he used computers. He had computer people talk to him all the time. Like, Mike was a really good computer guy. He knew what computers could do.
Computer meaning computer hardware.
Like hardware, software, all the pieces. And then he would have an idea about what could we do with this next that was grounded in reality. It wasn't like he was just finger painting on the wall and wishing somebody would interpret it like. So he had this interesting connection because now he wasn't a computer architect or a designer, but he had an intuition from the computers we had to what could happen.
And it's interesting you say intuition because it seems like he was pissing off a lot of engineers in his intuition about what can and can't be done. Those. Like the. What is all these stories about like floppy disks and all that kind of stuff like that.
Yeah. So in. In Steve, the first round, like he'd go into a lab and look at what's going on and hate it and. And fire people or ask somebody in the elevator what they're doing for Apple and, you know, not be happy when he came back. My impression was is he surrounded himself with this relatively small group of people.
Yes.
And didn't really interact outside of that as much. And then the joke was you'd see like somebody moving a prototype through the quad with a. With a black blanket over it. And that was because it was secret, you know, partly from Steve because they didn't want Steve to see it until it was ready.
Yeah. The dynamic with Jony I've and Steve is interesting. It's like you don't want to. He ruins as many ideas as he generates.
Yeah. Yeah.
It's a dangerous kind of line to.
Walk if you have a lot of ideas. Like Gordon Bell was famous for ideas. Right. And it wasn't that the percentage of good ideas was way higher than anybody else. It was. He had so many ideas and he was also good at talking to people about it and getting the filters right and, you know, seeing through stuff. Whereas Elon was like, hey, I want to build rockets. So. So Steve would hire a bunch of rocket guys and Elon would go read rocket manuals.
So Elon's a better engineer. A sense like. Or like, more like a love and passion for the manuals and the details. The craftsmanship, too, right? Well, I guess Steve had craftsmanship too, but of a different kind. What do you make of the. Just the standard for just a little longer. What do you make of, like, the anger and the passion and all that? The firing and the mood swings and the madness, being emotional and all that? That's Steve and I guess Elon too. So is that a bug or a feature?
It's a feature. So there's a graph which is y axis, productivity, X axis at zero. It's chaos. And infinity is complete order. So as you go from the, you know, the origin, as you improve order, you improve productivity.
Yeah.
And at some point, productivity peaks and then it goes back down again. Too much order, nothing can happen.
Yes, but the question is, is how close to the chaos is that?
No, no. Here's the thing is once you start moving, the direction or the force vector to drive you towards order is unstoppable. Oh, every organization will move to the place where their productivity is stymied by order.
So you need.
So the question is, who's the counter force? Because it also feels really good as you get more organized and productivity goes up. The organization feels it. They orient towards it. Right. They hired more people, they got more guys who can run process. You get bigger, and then inevitably the organization gets captured by the bureaucracy that manages all the processes. All right? And then humans really like that. And so if you just walk into a room and say, guys, love what you're doing, but I need you to have less order. If you don't have some force behind that, nothing will happen.
I. I can't tell you on how many levels. That's profound. So.
So that's why I'd say it's a feature. Now, could you be nicer about it? I don't know. I don't know any good examples of being nicer about it. Well, the funny thing is, to get stuff done, you need people who can manage stuff and manage people. Because humans are complicated. They need lots of care and feeding that you need to tell them they look nice and they're doing good stuff and pat them on the back. Right.
I don't know. You tell me. Is that needed do humans need that?
I had a friend, he started the magic group, and he said, I figured it out. You have to praise them before they do anything. I was waiting till they were done, and they were always mad at me. Now I tell them what a great job they're doing while they're doing it, but then you get stuck in that trap, because then when you're not doing something, how do you confront these people?
I think a lot of people that had trauma in their childhood would disagree with you. Successful people that you just first do the rough stuff and then be nice later. I don't know.
Okay. But, you know, engineering companies are full of adults who had all kinds of. Range of childhoods. You know, most people had okay childhoods.
Well, I don't know if lots of.
People only work for praise, which is weird.
You mean, like, everybody?
I'm not that interested in it, but.
Well, you.
You're.
You're probably looking for somebody's approval, even still.
Yeah. Maybe I should think about that.
Maybe somebody who's no longer with this kind of thing. I don't know.
I used to call up my dad and tell him what I was doing. He was. He was very excited about engineering and stuff.
You got his approval?
Yeah, a lot. I was lucky. Like, he. He decided I was smart and unusual as a kid, and that was okay when I was really young. So when I did poorly in school, I was dyslexic. I didn't read until I was third or fourth grade, and they didn't care. My parents were like, oh, he'll be fine. So I was lucky. That was cool.
Is he still with us? You miss him?
Sure. Yeah. He had Parkinson's and then cancer. His last 10 years were tough. And I killed him. Killing a man, like, that's hard.
The mind.
Well, it's pretty good. Parkinson's causes slow dementia, and the. The chemotherapy, I think, accelerated it, but it was like hallucinogenic dementia. So he was clever and funny and interesting and was. It was pretty unusual.
Do you remember conversations from that time? Like what? Do you have fond memories of the guy?
Yeah. Oh, yeah.
Anything come to mind?
A friend told me one time I could draw a computer on the whiteboard faster than anybody he'd ever met. And I said, you should meet my dad. Like, when I was a kid, he'd come home and say, I was driving by this bridge, and I was thinking about it, and he pulled out a piece of paper, and he'd draw the whole bridge. He was a mechanical engineer.
Yeah.
And he would just draw the whole thing. And then he would tell me about it and tell me how he would have changed it. And he had this, you know, idea that he could understand and conceive anything. And I just grew up with that. So that was natural. So if, you know, like when I interview people, I ask them to draw a picture of something they did on a whiteboard and it's really interesting. Like some people draw a little box, you know, and then they'll say, and this talks to this. And I'd be like, oh, this is frustrating. And I had this other guy come in one time, he says, well, I designed a floating point in this chip, but I'd really like to tell you how the whole thing works and then tell you how the floating point works inside of it. Do you mind if I do that? He covered two whiteboards in like 30 minutes. And I hired him like he was great.
Just craftsman. I mean, that's the craftsmanship to that.
Yeah. But also the mental agility to understand the whole thing.
Right.
Put the pieces in contacts, real view of the balance of how the design worked. Because if you don't understand it properly when you start to draw it, you'll fill up half the whiteboard with like a little piece of it. And you know, like your ability to lay it out in an understandable way takes a lot of understanding. So.
And be able to, so zoom into the detail and then zoom out to the picture.
Zoom out really fast.
What about the impossible thing? You see, your dad believed that you could do anything. That's a weird feature for a craftsman.
Yeah.
It seems that that echoes in your own behavior. Like that's, that's the.
Well, it's not that anybody can do anything right now. Right. It's that if you work at it, you can get better at it and there might not be a limit. And they did funny things. Like he always wanted to play piano. So at the end of his life he started playing the piano when he had Parkinson's. And he was terrible. But he thought if he really worked at it in this life, maybe the next life he'd be better at it.
He might be onto something.
Yeah, he enjoyed doing it. Yeah. So it's pretty funny.
Do you think the perfect is the enemy of the good in hardware and software engineering? It's like we were talking about JavaScript a little bit and the messiness of the 10 day building process.
Yeah. You know, creative tension. Right. So creative tension is you have two different ideas that you can't do both. Right. And, and, but the fact that you Want to do both causes you to go try to solve that problem. That's the creative part. So if you're building computers, like some people say, we have the schedule, and anything that doesn't fit in the schedule we can't do. So they throw out the perfect because they have a schedule. I hate that. Then there's other people who say we need to get this perfectly right, and no matter what, more people, more money. And there's a really clear idea about what you want. And some people are really good at articulating it. So let's call that the perfect. Yeah, yeah. All right. But that's also terrible because they never ship anything. They never hit any goals. So now you have the. Now you have your framework.
Yes.
You can't throw out stuff because you can't get it done today, because maybe you get it done tomorrow or the next project. Right. You can't. So you have to. I work with a guy that I really like working with, but he over filters his ideas over filters. He'd start thinking about something, and as soon as he figured out what was wrong with it, he'd throw it out. And then I start thinking about it and, you know, you come up with an idea, and then you find out what's wrong with it, and then you give it a little time to set, because sometimes you figure out how to tweak it or maybe that idea helps some other idea. So idea generation is really funny. So you have to give your ideas space. Spaciousness of mind is key, but you also have to execute programs and get shit done. And then it turns out computer engineering is fun because it takes 100 people to build a computer, 200 to 300, whatever the number is. And people are so variable about, you know, temperament and, you know, skill sets and stuff, that in a big organization, you find the people who love the perfect ideas and the people that want to get stuff done yesterday. And people like to come up with ideas, and people like to, let's say, shoot down ideas. And it takes the whole. It takes a large group of people.
Some are good at generating ideas, some are good at filtering ideas. And then they all. In that giant mess, you're somehow. I guess the goal is for that giant mess of people to find the perfect path through the. The tension, the creative tension. But, like, how do you know when you said there's some people good at articulating what perfect looks like, what a good design is? Like, if you're sitting in a. In a room and you have a set of ideas about, like, how to design a better processor. How do you know this is something special here? This is a good idea. Let's try this.
Have you ever brainstormed idea with a couple of people that were really smart and you kind of go into it and you don't quite understand it, and you're working on it, and then you start talking about it, putting it on the whiteboard, maybe it takes days or weeks, and then your brain start to kind of synchronize. It's really weird. Like, you start to see what each other is thinking and it starts to work. Like, you can see work. Like, my talent in computer design is I can see how computers work in my head, like, really well. And I know other people can do that too. And when you're working with people that can do that, like, it is kind of an amazing experience. And then. And every once in a while you get to that place and then you find the flaw, which is kind of funny because you can fool yourself, but.
The two of you kind of drifted along into a direction that was useless.
Yeah, that happens too. Like, you have to. Because, you know, the nice thing about computer design is always reduction practice. Like, you come up with your good ideas, and I know some architects who really love ideas, and then they work on them and they put it on the shelf and they go work on the next idea and put it on the shelf. They never reduce it to practice, so they find out what's good and bad. Because almost every time I've done something really new, by the time it's done, like, the good parts are good, but I know all the flaws.
Yeah. Would you say your career, just your own experience, is your career defined by mostly by flaws or by successes?
Again, there's great tension between knows. If you haven't tried hard. Yeah. Right. And done something new. Right. Then you're. You're not going to be facing the challenges when you build it. Then you find out all the problems with it and.
But when you look back, do you see problems or. Okay.
Oh, when I look back, what do you. I think earlier in my career.
Yeah.
Like, EV5 was the second alpha chip. I was so embarrassed about the mistakes, I could barely talk about it. And it was in the Guinness Book of Worlds records, and it was the fastest processor on the planet.
Yeah.
So it was. And at some point I realized that was really a bad mental framework to deal with, like, doing something new. We did a bunch of new things and some worked out great and some were bad, and we learned a lot from it. And then the next one we learned a lot that also EV6 also had some really cool things in it. I think the proportion of good stuff went up, but it had a couple of fatal flaws in it that were painful. And then.
Yeah, you learned to channel the pain into like, pride.
Not pride, really, just realization about how the world works or how that kind of idea set works.
Life is suffering. That's the reality. What?
No, it's not. I know the Buddha said that and a couple other people are stuck on it. No, it's, you know, there's this kind of weird combination of good and bad, light and darkness that you have to tolerate and deal with. Yeah, there's definitely lots of suffering in the world.
Depends on the perspective. It seems like there's way more darkness, but that makes the light part really nice. What computing hardware or just any kind of even software design are you do you find beautiful from your own work, from other people's work that you're just. We were just talking about the, the battleground of flaws and mistakes and errors, but things that were just beautifully done. Is there something that pops to mind?
Well, when things are beautifully done, usually there's a well thought out set of abstraction layers.
So the whole thing works in unison. Nicely.
Yes. And when I say abstraction layer, that means two different components. When they work together, they work independently. They don't have to know what the other one is doing.
So that decoupling.
Yeah, so the famous one was the network stack. Like there's a seven layer network stack, you know, data transportation protocol and all the layers. And the innovation was, is when they really got that right, because networks before that didn't define those very well. The layers could innovate independently and occasionally the layer boundary, the interface would be upgraded and that let the design space breathe. You could do something new in layer seven without having to worry about how layer four worked. And so good design does that, and you see it in processor designs. When we did the Zen design at amd, we made several components very modular. And, you know, my insistence at the top was I wanted all the interfaces defined before we wrote the RTL for the pieces. One of the verification leads said, if we do this right, I can test the pieces so well independently. When we put it together, we won't find all these interaction bugs because the floating point knows how the cache works. And I was a little skeptical, but he was mostly right that the modularity of the design greatly improved the quality.
Is that universally true in general, would you say about good designs? The modularity is like, usually.
Well, we talked about this before. Humans are only so smart and we're not getting any smarter. Right. But the complexity of things is going up. Yeah. So, you know, a beautiful design can't be bigger than the person doing it. It's just, you know, their piece of it. Like, the odds of you doing a really beautiful design of something that's way too hard for you is low. Right. If it's way too simple for you, it's not that interesting. It's like, well, anybody could do that. But when you get the right match of your, your expertise and, you know, mental power to the right design size, that's cool, but that's not big enough to make a meaningful impact in the world. So now you have to have some framework to design the pieces so that the whole thing is big and harmonious. But when you put it together, it's sufficiently interesting to be used. So that's like a beautiful design.
Matching the limits of that human cognitive capacity to the module you can create and creating a nice interface between those modules. And thereby, do you think there's a limit to the kind of beautiful, complex systems we can build with this kind of modular design? It's like if we build increasingly more complicated. You can think of the Internet. Okay, let's scale it down. You can think of social network like Twitter as one computing system, but those are little modules, but it's built on.
So many components, nobody at Twitter even understands. So if an alien showed up and looked at Twitter, you wouldn't just see Twitter as a beautiful, simple thing that everybody uses, which is really big. You would see the network runs on the fiber optics, the data's transported, the computers. The whole thing is so bloody complicated, nobody at Twitter understands it.
And so I think that's what the alien would see. So, yeah, if an alien showed up and looked at Twitter or looked at the various different networked systems that you can see on Earth.
So imagine they were really smart, that could comprehend the whole thing, and then they sort of, you know, evaluated a human and thought, this is really interesting. No human on this planet comprehends the system they built.
No individual. Well, would they even see individual humans? That's like, we humans are very human centric, entity centric. And so we think of us as the organ, as the central organism and the networks as just the connection of organisms. But from a perspective of inhale it, from an outside perspective, it seems like.
Yeah, I get it. Where the ants. And they'd see the ant colony.
The ant colony, yeah. Or the result, the production of the ant colony, which is like Cities and yeah, it's, it's. In that sense, humans are pretty impressive. The modularity that we're able to and the, and how robust we are to noise and mutation, all that kind of stuff.
Well, that's because it's stress tested all the time. Yeah, you know, you build all these cities with buildings and you get earthquakes occasionally and some wars, earthquakes, viruses, every once in a while, changes in business plans or like shipping or something like, as long as it's all stress tested, then it keeps adapting to the situation. So that's a curious phenomena.
Well, let's talk about Moore's Law a little bit at the broad view of Moore's Law, where it's just exponential improvement of computing capability. Like OpenAI, for example, recently published this kind of papers looking at the exponential improvement in the training efficiency of neural networks for like imagenet and all that kind of stuff. We just got better on this, this is purely software side, just figuring out better tricks and algorithms for training neural networks. And that seems to be improving significantly faster than the Moore's Law prediction. You know, so that's. In the software space. They. What do you think? If Moore's Law continues or if the general version of Moore's Law continues, do you think that comes mostly from the hardware, from the software? Some mix of the two? Some interesting? Totally. So not, not the reduction of the size of the transistor kind of thing, but more in the, in the totally interesting kinds of innovations in the hardware space, all that kind of stuff.
Well, there's like half a dozen things going on in that graph. One is there's initial innovations that had a lot of headroom to be exploited. So the efficiency of the networks has improved dramatically and then the decomposability of those. And they started running on one computer, then multiple computers and multiple GPUs and then arrays of GPUs, and they're up to thousands. And at some point, so it's sort of like they were going from like a single computer application to a thousand computer application. So that's not really a Moore's Law thing, that's an independent vector. How many computers can I put on this problem? Because the computers themselves are getting better on like a Moore's Law rate, but their ability to go from 1 to 10 to 100 to 1000 was something. And then multiplied by the amount of computes it took to resolve, like Alexnet to resnet to Transformers, it's been quite, you know, steady improvements.
But those are like S curves, aren't they? That's the Exactly. Kind of S curves that are underlying Moore's Law from the very beginning. So what's the most productive rich source of S curves in the future, do you think? Is it hardware or is it software?
So hardware is going to move along relatively slowly, like, you know, double performance every two years. There's still.
I like how you call that slow.
Yeah, that's the slow version, the snail's pace of Moore's Law. Maybe we should trademark that one. Whereas the scaling by number of computers can go much faster. I'm sure at some point Google had their initial search engine was running on a laptop, and at some point they really worked on scaling that. And then they factored the indexer from this piece and this piece and this piece, and they spread the data on more and more things, and they did a dozen innovations. But as they scaled up the number of computers on that, it kept breaking, finding new bottlenecks in their software and their schedulers and made them rethink. It seems insane to do a scheduler across a thousand computers who schedule parts of it and then send the results to one computer. But if you want to schedule a million searches, that makes perfect sense. So the scaling by just quantity is probably the richest thing. But then as you squale quantity, like a network that Was great on 100 computers may be completely the wrong one. You may pick a network that's 10 times slower on 10,000 computers, like per computer. But if you go from 100 to 10,000, that's 100 times. So that's one of the things that happened when we did Internet scaling is the efficiency went down, not up. The future of computing is inefficiency. Not efficiency, but scale.
Inefficient scale.
It's scaling faster than inefficiency bites you. And as long as there's dollar value there, like, scaling costs lots of money. But Google showed, Facebook showed, everybody showed that the scale was where the money.
Was at, so it was worth it financially. Do you think? Is it possible that, like, basically the entirety of Earth will be like a computing surface? Like this table will be doing computing, this hedgehog will be doing computing. Like, everything really inefficient. Dome computing will be like science fiction books.
They call it Computronium.
Computronium.
We turn everything into computing. Well, most of the elements aren't very good for anything. Like, you're not going to make a computer out of iron. Like, you know, silicon and carbon have, like, nice structures. You know, we'll see what you can do with the rest of it. People Talk about, well, maybe you can turn the sun into computer. But it's, it's hydrogen and a little bit of helium.
So what I mean is more like actually just adding computers to everything.
Oh, okay. I thought you're just converting all the mass of the universe into a computer.
No, no, no. So not using.
It'd be ironic from the simulation point of view. It's like the simulator build mass to simulate, like.
Yeah, I mean, yeah. So I mean ultimately this is all heading towards a simulation.
Yes. Well, I think I might have told you this story. At Tesla, they were deciding, so they want to measure the current coming out of the battery and they decided between putting the resistor in there and putting a computer with a sensor in there. And the computer was faster than the computer I worked on in 1982. And we chose the computer because it was cheaper than the resistor. So sure, this hedgehog, it costs $13 and we can put an AI that's as smart as you in there. For five bucks it'll have one. So computers will be everywhere.
I was hoping it wouldn't be smarter than me because, well, everything's going to.
Be smarter than you.
But you were saying it's inefficient. I thought it was better to have a lot of dumb things.
Well, Moore's law will slowly compact that stuff.
So even the dumb things will be smarter than us.
The dumb things are going to be smart or are they going to be smart enough to talk to something that's really smart? You know, it's like, well, just remember, like a big computer chip. It's like an inch by an inch and 40 microns thick. It doesn't take very much, very many atoms to make a high power computer and 10,000 of them can fit in the shoebox. But you have the cooling and power problems. But people are working on that.
But they still can't write compelling poetry or music or understand what love is or have a fear of mortality. So. So we're still winning.
Neither can most of humanity. So.
Well, they can write books about it. So. But, but speaking about this, this walk along the path of innovation towards the dumb things being smarter than humans. You are now the CTO of 10 storen as of two months ago. They build hardware for deep learning. How do you build scalable and efficient deep learning? This is such a fascinating space.
Yeah, so it's interesting. So up until recently I thought there was two kinds of computers. There are serial computers that run like C programs and then there's parallel computers. So the way I think about it is parallel. Computers have given parallelism. Like GPUs are great because you have a million pixels. And modern GPUs run a program on every pixel. They call it a shader program. Right. Or like finite element analysis. You build something, you make this into little tiny chunks. You give each chunk to a computer, so you're given all these chunks. You have parallelism like that. But most C programs, you write this linear narrative and you have to make it go fast. To make it go fast, you predict all the branches, all the data fetches, and you run that more in parallel. But that's found parallelism. AI is. I'm still trying to decide how fundamental this is. It's a given parallelism problem. But the way people describe the neural networks and then how they write them in Pytorch, it makes graphs.
Yeah. That might be fundamentally different than the GPU kind of parallelism.
Yeah, it might be. Because when you run the GPU program on all the pixels you're running depends. This group of pixels say it's background blue and it runs a really simple program. This pixel is some patch of your face. So you have some really interesting shader program to give you the impression of translucency. But the pixels themselves don't talk to each other. There's no graph. Right. So you do the image and then you do the next image, and you do the next image. And you run 8 million pixels, 8 million programs every time. And modern GPUs have like 6,000 thread engines in them. So to get 8 million pixels each, one runs a program on 10 or 20 pixels, and that's how they work. But there's no graph.
But you think graph might be a totally new way to think about hardware.
So Raja Ghadori and I have been having this conversation about given versus found parallelism. And then the kind of walk is we got more transistors, like computers way back when, did stuff on scalar data. Then we did it on vector data. Famous vector machines. Now we're making computers that operate on matrices. And then the category we said was next was spatial. Like, imagine you have so much data that you want to do the compute on this data. And then when it's done, it says, send the result to this pile of data. Run some software on that. And it's better to think about it spatially than to move all the data to a central processor and do all the work.
So spatially you mean moving in the space of data as opposed to moving the data?
You have a petabyte data Space spread across some huge array of computers. And when you do a computation somewhere, you send the result of that computation or maybe a pointer to the next program to some other piece of data and do it. But I think a better word might be graph. And all the AI neural networks are graphs. Do some computations and the result here, do another computation, do a data transformation, do a merging, do a pooling, do another computation.
Is it possible to compress and say how we make this thing efficient, this whole process efficient? There's different.
So first, the fundamental elements in the graphs are things like matrix multiplies, convolutions, data manipulations and data movements. So GPUs emulate those things with their little singles, basically running the single threaded program. And then there's a, you know, Nvidia calls it a warp, where they group a bunch of programs that are similar together. So for efficiency and instruction use. And then at a higher level, you kind of, you take this graph and you say this part of the graph is a matrix multiplier which runs on these 32 threads. But the model at the bottom was built for running programs on pixels, not executing graphs.
So it's emulation ultimately. So is it possible to build something that natively runs graphs?
Yes. So that's what 10 Storm did.
So where are we on that in the history of that effort? Are we in the early days?
Yeah, I think so. 10 stored by a friend of mine, Labija Bajik, and I, I was his first investor. So I've been, you know, kind of following him and talking to him about it for years. And in the fall when I was considering things to do, I decided, you know, the. We, we held a conference last year with a friend, organized it and, and we, we wanted to bring in thinkers. And two of the people were Andre Carpassi and Chris Lattner. And Andre gave this Talk, it's on YouTube called Software 2.0, which I think is great. Which is we went from programmed computers where you write programs to data program computers. You know, like the future is, you know, of software is data programs, the networks. And I think that's true. And then Chris has been work, he worked on llvm, the low level virtual machine, which became the intermediate representation of.
