#24: The "faster than OpenAI" AI chip

So this is going to be a fun one. Everybody in San Francisco is telling me about this company. It's like, oh, grok runs my AIs way faster than anything else. Two times faster, I heard, than even the new model at OpenAI. So I'm real happy to have the CEO of Grok. That's Grok with a Q. There's another Groq somewhere, and we'll maybe talk about that, too. So who are you?

Me? I'm. I'm Jonathan Ross. I'm CEO and founder of Grok with a Q. And wait, only two times faster.

All right, well, you know, early tests, I don't know.

At least.

At least two times faster, so. Which is a huge, huge amount of deal. So let's start at 30,000ft. For people who don't understand the AI world that well, you're not. You're building a new kind of chip that runs AIs at inferencing time. In other words, at runtime, you're not building a chip to build the models. Right. You're not going to war with Nvidia trying to take away market share from Nvidia. You're doing something new, right? Yeah, we're a little more about what you're doing.

Well, first of all, we're very friendly with Nvidia. In fact, we have an unofficial deal where if you buy a Grok rack, they will put you to the front of the line for whatever the latest GPUs are.

Oh, that's cool.

Just let your Nvidia sales rep know that you're buying something from Grok. It's very helpful.

And silicon is one thing. What's the breakthrough here? Why are you able to run at inferencing time so much faster than anybody else?

Well, we have a completely different architecture. And it's a little bit like saying, why is it that GPUs can run much faster than CPUs? They were more fit for purpose. GPUs are really good at parallel. CPUs are really good at sequential. Groq is uniquely good at both. So if you have a large amount of compute that you have to perform, but in a very short period of time, we can do that faster. And for anything generative AI, there's a sequential component to it. You can't predict the 100th word until you predicted the 99th and the 98th and so on. And so you need that sequential component in order to get the speed.

And.

I mean, how does know how the math works?

Yeah, I want to know how it really works. What's the get nerdy for? For the people who really care about what's going on under the hood. What, what, what makes your chips, what was the breakthrough? Because obviously you had a breakthrough, you put it on a PowerPoint deck or something like that. You went to venture capitalists and got funding to do this. But what was the breakthrough?

We completely reinvented the entire stack, so it wasn't one thing. You know, when I get asked, what's your secret sauce? I say, we don't have secret sauce. We have 11 herbs and spices. And the reason is there's six different places in the stack where you're going to have a bottleneck. The first is on the chip itself. That's obvious. That's what everyone's working on. Then the system, Nvidia's figured that out. Not too many other people have. And then the compiler, which you can throw a lot of effort at, but you're not going to get great performance unless you do some of the stuff we've done. But then there's also the runtime, the orchestration layer and the networking, which all have to be completely reinvented or they're going to be the bottleneck. The way to think about it is Jeff Dean, famous computer scientist at Google. He has this document that he's published which is engineering number or numbers that every engineer should know. And the speed of a lot of this stuff hasn't really changed in the 10 years since he's published it. So if you are using dram, yeah, there's a little more throughput, but it's not lower latency. If you're using dis, hasn't really improved, so Flash hasn't really improved that much. You get a little more throughput, but not lower latency. And for inference, unlike training, latency matters. This is the whole sequential component. What we've done is we shifted left. We refer to DRAM as disk because it's so slow. If we were to run anything using dram, we would be hobbled in our performance. So we use DRAM like disk. We do have it, actually. We have almost, I think, a petabyte of DRAM in our cloud today, but. Yeah, exactly. But we use it to store things that are infrequently used, like in the course of milliseconds, not microseconds or nanoseconds. The sram, which is the memory inside of the chip, is where we do all of our computation. And that's really the trick. It's almost like, you know, everyone thinks about Google as a web service and whatnot. And what they don't realize is every time they type in a query and they hit enter, that query is processed by tens of thousands of servers for that single query. And that's where that speed comes from. But each one is used for a very short period of time and that's why it's economical. Well, we do the same thing, but we do it for AI.

That's awesome. Does this affect the economics? Because a lot of developers are out there are like, okay, do I host the back end on, on OpenAI using chat GPT or do I do use some other, do I use AWS to host this stuff? What's the economics comparables?

So there's a massive impact to the economics. If you are using a gpu, you have to buy, it uses hbm, which is a very expensive type of memory, it uses an interposer which is expensive. And then you, you've got very expensive interconnect and all that. We've simplified the system so most of the system isn't even there. And you don't. Well, we don't have to pay for it. The prices that we have published on our website, those are prices that we're going to be profitable at. So, and, and happy to support those prices for a long period of time, especially at, let's just say we may also be a little more disruptive in the future. But the, the point here is when you simplify the system, you reduce the cost pretty dramatically. But that's not the real issue. The real issue is the power. So if you look at the cost of running a gpu, just the power and OPEX cost of that GPU is higher than our cost when you include our opex plus our capex. So in other words, GPUs could be given away for free and it would still be more expensive to run inference on a GPU than on GROK at the full speed because we're built for that speed. Now we also are going to be getting a little bit faster, just a little bit faster over time. And so we're going to continue to get faster while remaining inexpensive.

This, this has huge implications across our economy, right? Because we're soon going to be running humanoid robots on these things. We're running autonomous cars using this, this kind of AI technology. And obviously we're getting new kinds of chat bots that we talk to, you know, that bring huge new utility and a whole bunch of new developer class that's building all sorts of interesting stuff using various models whether it's OpenAI's model or llama or something else. Where do you. Since you're in the middle of all this, all these guys are probably coming to you going, hey, can I use your inferencing to run our thing? Right. What are you seeing? Let's go back up to 100,000ft. What are you seeing happening in the AI world? And what should we pay attention to over the next year? This is 2024, right?

Well, funny you mentioned 100,000ft, because in 10 weeks, we've gone from 10 developers in our closed beta to over 175,000 developers on our site. And given your background, you know what that means?

Yeah, that's fast. You're faster than ICQ grew back in the day. Oh, ICQ went from nobody to 65,000 user in six weeks and was the number one social software of. Of 1996. So I use that as my benchmark. You know, do you grow faster than an icq? Oh, okay. You're doing fine.

And developers are harder to get than, of course, just end users. And that's developers, because we've had, I think, more like a million people use our chat site. But that's just the developers.

Yeah, so what?

Yeah, so, yeah.

What's happening from your point of view in the AI industry? And what should we be paying attention to here?

Yeah, so first off, people are still trying to figure it out. We see a lot of very similar ideas. So as an example, I think there's been 15 different versions of Perplexity that have been built on top of us, and that's just the ones that we've seen. But there are some really novel things that are interesting and you could see them evolving into a product, but they're not there yet. Like, one of my favorites is Infinite Wikipedia. And because of the speed it generates a Wikipedia article, you click one of the links in it and then it instantly generates the next link. So you can just keep going forever, as if Wikipedia wasn't bad enough and could keep you going for a long time. But the more interesting stuff, the way that I look at it, is first you're going to have a brainstorming partner where you don't necessarily trust the results, but it's creative and it's helping you. Then the next step is being able to retrieve information with high fidelity. We're starting to see people making progress on that. The next step will be validating results from human beings where, yes, I really trust that this is good, because an AI said that this was good. Yeah, it's going to check your work, it's going to make you more effective. Think about all the time that you hate spending debugging your code. Just imagine that the AI can just tell you that's where the bug is. And if you hit enter, there's probably no bug.

My psychiatrist uses AI to listen to our therapy sessions and it catches mistakes that she, it catches things that she doesn't catch. Right. So you're, we're right on.

We're going to have. I think there's an element of when an AI tells you you made a mistake, it, you're less defensive than when a human being does. And when I, when I first started programming, my ability to reason improved pretty dramatically because I had something all day long telling me that I was wrong and I believed it. I mean, I think when I first started programming, the first couple of times I got an error, my thought was actually, wait a second, I didn't have an error. What do you mean? And then eventually I just learned, oh, anytime it says I have an error, I have an error. And it changed my thinking. Right. I think AI is going to do that in a lot more spaces. Then finally we're going to start allowing it to make some very simple decisions for us. And that's where we need to pay a lot of attention because the most important thing is to not give over our autonomy or our decision making authority. But there are only a finite number of decisions that a human being can make in a day. Otherwise you have decision fatigue. And if we can very clearly bucket the decisions that it's okay for an AI to make and the ones where it's not, we can get a lot more done.

Yeah, that's crazy stuff. My last question before I start pulling up questions from my followers. What does this mean for consumer electronics? Because, and here's where I'm going, Apple, I'm, I'm talking to you on an Apple Macintosh which has an M1 processor. And in that processor there's 21% is AI inferencing. Sort of like you, right? It's not as fast, but I want, I want Grok in my local Macintosh so, so that I can run all my AIs really, really fast locally. Right. Maybe it put you in a robot, and so my robot will be interacting with me very, very quickly. Is that, is that sort of on the road map for you guys? Or are you always going to be like a thing that we hire and a service that's in the data center?

We may have had some very large companies ask us very similar questions. So without getting into details of what we are and are not going to do, our architecture is synchronous and deterministic. And when you think about robotics, think about it from this point of view. Imagine that you have VR goggles on. And those VR goggles have a camera with a feed directly into VR goggles. So you're seeing what the camera sees. Exactly. You got it. All right, now add one thing to those VR goggles, which is we want to add a little knob to the side which allows you to set the delay from what you're seeing and you can set it to 0, 1 second, 3 seconds. Now I want you to imagine chopping carrots with a one second delay or a three second delay. It's getting a little scary. Now imagine if someone throws you. Yes, you'll ruin the carrots. Don't do that.

But.

But it gets harder the longer the delay, right?

Yeah.

And just to understand that more viscerally, imagine a kid throws you a ball with a very short one second delay. And they're far enough away. It's not that hard. But if it's three seconds, you have to predict that the kid is even going to throw the ball. It becomes super non linear. So the faster your prediction, the more linear, the fewer ops. And this is something that people don't often understand. The faster that you run, the more powerful each op is. Right. It's not just the bit width, although the bit width does matter. And it's starting to matter a lot since we're past chinchilla optimal. You want to, you want to geek out. We'll geek out on that in a minute. But, but the speed changes the value of the computer. And when it comes to robotics, oftentimes they have to make the arms very heavy. They have to make them heavy because it makes it easier to control, because there's less pushback from the items that they're manipulating. But if you can very quickly respond, then you can make the arm lighter, use less energy and so on. And then as for phones, this is something that people very often conflate. There's a difference between energy and power. Power is the amount of energy per second, right? It's energy over time that's being delivered, whereas energy is the total amount used to complete a task. Mobile phones are lower power, but they're actually worse for energy. And there's a couple of reasons why. Number one, first of all, you have to charge and discharge a battery. That's inefficient. That's the simplest but the second is, and most people forget this, most phone chips are off almost all the time waiting to do something. A modern fab requires a gigawatt of power to run. Data centers aren't that big yet. So that chip that's being built, an enormous amount of power went into building that chip. An enormous amount of carbon was released. And if you're not fully utilizing it, it's actually worse for the environment. So it's like transporting tons of coal across a city using a moped versus a freight train. The moped will be lower power, the freight train will be lower energy, and you want to use that freight train. Data centers are actually much more energy efficient for things that are generative, AI, even image. But let's take language for a moment. Two bytes of data in, 180 billion operations, two bytes out. It's so skewed that it is much more energy efficient to transport the data to the computer, compute it and send it out. We're also much faster than doing it on the phone because just like when you do a Google search and they use tens of thousands of chips, when you use grok, we will use hundreds, thousands or more chips to very quickly get you an answer with much lower energy used for that answer, hence our much lower cost. And then we send it to you faster than it could have ever been done locally.

So this means your augmented reality glasses, your robots, your. Well, your autonomous car doesn't care about power. So but you do have to be aware of it because you use less power, it goes more miles. Right. Which means you're making more money with a robotaxi in the future, right?

Yeah, autonomous vehicles are different. You're going to be putting the compute in the car. And the reason is you need a guaranteed real time experience. It's not good enough to be better with four nines or five nines, it has to be all the nines. Right. And that's different. Now that said, route planning, actually in self driving cars today, whenever it gets stuck and it can't figure it out, a human assist driver gets in like not physically but like remotely will control it and get it out of that. So there's still this remote component, but also when you're speaking to it, the voice controls, they're probably not going to be local because you're going to want the compute to be able to deliver that quickly. Now what you can do is you can have a backup in the car which will use more energy. When it kicks in, it won't be as high quality, it'll make more mistakes in understanding you, but it will be good enough versus not having a connection.

Yeah, what a world. How did you get into this? Why did you choose semiconductors for your career? You're a smart dude, you could run a bank or do anything.

Started with a lunch.

Started with what? Lunch.

Lunch. Like eating lunch with smart people. So I was working in the Google, Google New York City office. I'd had a little bit of FPGA experience from finance, but I was self taught there. I never took a course in it. And the speech recognition team at Google had trained at this time the first speech recognition model that outperformed human beings. And at Google you have lunch with people. And so we were having lunch and they're, they're complaining that the social thing, Google plus is what it was later called circles or whatever. I forget what the internal was. I think it was like Emerald Sea or something. It was something else internally. But it had taken all of the compute and so they couldn't launch this. And it was amazing because I remember back then. Do you remember how bad voice dictation software was?

Yeah. Kai Fu Lee showed me NLP at Microsoft in the early days. It couldn't understand me in a quiet conference room. Now we have, you know, things like Whisper. You can be talking to it in a rock concert. It's crazy.

That's right. And so back in 2011, I remember I was in the lunchroom and they were showing me and it was noisy, there was something going on and I was saying really complicated words and it was getting it like that. I'm like, we've got to get this out there. So I started working on an FPGA project which is reprogrammable chip. It's when you're not fully committed to the design yet, you can iterate on it. And it got some pretty good results. And Jeff Dean, who's kind of the goat at Google, he ended up doing a presentation to the leadership team which was good news, AI works, bad news, we can't afford it. However we can build these chips. We were going to do it as an fpga. Jeff Dean's like, nah, let's build a chip. And we're like, how hard could that be? It's really hard. But we did it. And fortunately we didn't know how hard it was. So we all signed up like a bunch of suckers and we just made it work.

I hear this story over and over again in the industries. Oh, Steve Jobs talked me into doing something really hard. And in this case it was Jeff Dean. Wow. Where do we take this conversation? Because at one level you're a very complex business, but at another level, it's sort of a simple business, isn't it?

Yeah. So we could take this a little bit off road. So how do we get here? I mean, there was an element of, we always wanted to do something in cloud. That's my background, distributed systems. You know, this is why we weren't afraid of using a large number of chips. Like, that's just what you do when you're a cloud provider. You use a large number of chips, get your cost down and everything. Right. But we tried selling that and because we needed money in the interim, up until we got into a cloud business and no one wanted it and we just, we couldn't understand because we're like, it's better. And it reminded me a little of the story of the kid who traded three dimes for two quarters because, you know, like, or, you know, or three quarter or three dimes for two quarters, because three dimes is more. Right. And so everyone's just fixated on, well, but I can do this on a smaller number of chips. It's like, yeah, I could run any of these models on an Arduino. It wouldn't be cost effective and it wouldn't be fast. Right. And so we just had to build all the way to having our own API that developers could use. And the funny part was the beginning of this year, we had a couple of conversations with some VCs, and there, these, these two VCs in particular, they really didn't get it. They're all software. They didn't understand. They're like. And I'm like, okay. And this was, remember the beginning of the year, I'm like, you should meet with us anyway, because we're going to be the company that every single one of your startups is building on by the end of the year. And so that was a little bit, you know, it was a little bit of a. It seemed like a stretch of a prediction at the time. But if you understand why latency matters in all of these cloud providers. And like, you know, the Spotify movie if you watch, or a TV show on Netflix if you watch it, it's all about getting the latency down. Obsessed with this. And that's because that drives user behavior more than anything else. But we were getting people saying, why would you need this to be faster than you can read? And we'd be like, why do you need Google searches faster than you can read? Because it's the Only way you'll ever use it. If it comes out slow, you won't use it.

Yeah, people don't get the future until long after it arrives. Most of them, people like you are always ahead, but the rest of us struggle to catch up, you know, well.

And, but, but when it's this once tangible, right. Once you can feel it and then.

You'Re like years ago you talked to normal people about AI. They didn't, they don't, didn't know why it's important. You know, they didn't have a concept. You're going to be talking to this thing and it's going to be talking back soon and, and you're going to be having it write college essays and do all the stuff we're doing with AI now. Right.

I was at a, I was at a big conference about a year and a half ago. It was, it was, I think in a September. It was the September before GPT3.3 was released. And it was funny because someone, someone from Anthropic was there and they demoed a chatbot and I was just looking around and no one was reacting.

Yeah.

And I'm like, do people not understand what they're seeing? And, and it took me a little while to realize, oh, if you're not the one typing and it's responding directly to you, you don't understand that this isn't, can't. Like there's a cognitive dissonance. So when we started doing demos of things running fast on us, we would always participate, someone from the audience. And at World Government Summit we did a voice demo like the ones that were just done, but this was done last year and we went through the demo and people were reacting a little bit, but not much. And then we had a reporter who is well known in the region come on stage and ask a question and answered and someone went like this.

Yeah.

Literally, like it wasn't that moment they realized it was responding to the questions and it wasn't canned.

Yeah, AI is a crazy thing. I'm there. Some of my readers are asking questions like, are you ever going to do a model building computer? Like, you know what a lot of these AI companies are using Nvidia for to build their models.

Right.

Mark Zuckerberg has 300,000 H100 cards in data centers. Right.

Well, we think that GPUs are really good at training. We're not going to compete with them on that. I mean you need parallel compute, you do not need a sequential element. And frankly, the more you use those GPUs for training, the more you free up using our compute for inference, which is what we're good at, and vice versa. In fact, there are a couple of people now who were getting ready to get a bunch of GPUs who are still going to get them because now they're going to use them all for training and allocation is hard to get. And then there's a couple who've already got some GPUs that they're using for inference, who intend now to repurpose them for training, and then they want to use us for all their inference.

Yeah.

And what I think is going to happen, because I've seen this story play out a bunch of times, the more inference compute people get, the more it pays off to train more, the more valuable the models get, the more people need inference. And so, for example, when Llama 3 came out, our developer growth spiked because the quality was better. The model was therefore better. There was more reason to build applications on top of it. Right. But that then increases the demand for more training. So while I know Nvidia probably isn't going to see it this way, GROK is probably going to be the best thing that's ever happened for their stock because the demand for training is going to be more insatiable and they don't really serve inference cost effectively. And so that was going to be like the block on that. So I actually think their stock is going to be more valuable.

Sounds like a good pitch. Maybe you get acquired someday soon, right? By one of these big, big months.

We're not looking. Yeah, we're not looking to get acquired. Always happy to have conversations, but our focus is on how do we.

Every entrepreneur says that, I'm going all the way, I'm going public, I don't want to be, I don't want to be. And then a week later, they're getting acquired by some big company for some pilot cash.

We get asked this a lot and our response is, is always the same. If that's really your interest, why don't we just work together on something and see if we can even work together first? Because the realization is what we're doing in the way that we're doing it, there's value for us to be across the entire ecosystem. And once they start to see that, they start to realize, okay, we still want this to succeed. But you're right, it's actually better if we don't own it. And that's. We always push that. That's a very clear position of ours.

Yeah. Mr. Y asked how can soon can you do On Prem inferencing? And the, the reason it a lot of businesses don't want to give all their data up to the cloud. Up to some cloud, even your cloud. Right. And it doesn't want to give it up to models where you're not quite sure. Does Mark Zuckerberg get to see that data? Do they use that data for training that future model? And all of a sudden intellectual property is leaking out and stuff like that. Right. That, that's the fear. So there's an impulse of, oh, let's run all of our inferencing inside our own firewall at our own business. We where we can control everything and watch to make sure nothing leaves.

All right, so first off, we do offer some on Prem solutions. However, before we even get there, we don't store any data past I think one hour. And the reason is we're not training models. We don't create the models, we provide the compute, we don't create the applications, we provide the compute. That's it. And so we have no need for that data other than certain things like how, how large was the input and output because we need to bill for that and things like that. Right. And so we don't have your data. We couldn't look at it past an hour if we wanted to. Now we do need to maintain it for about an hour because, you know, people try and do nefarious things and all that and we need to be able to see what's going on, but we don't, we just don't store data. And then the last part of that is on prem, we have a couple of people who want on Prem solutions, but there's also managed by us. So if you want to have hardware that is completely isolated, you can also just work with us. We will dedicate that hardware to you. No one else will be on it. It's not as cost effective because you have to pay for the whole usage. We're very good at load balancing. But finally we do have on Prem solutions for those who really, really don't want to send anything outside. But we'll meet customers wherever they are. We also provide the software layer, our cloud layer. So even if you have it on Prem, you just make API calls. So it's the same whether it's in cloud or on prem. You don't have to do a bunch of gnarly driver hacking and things like that.

Yeah, I think we already covered this. Nick was asking about local inferencing engines in, in driving situations, in Autonomous vehicles. Are you going to build a inferencing computer for some, you know, new, some car company, General Motors or something like that to put autonomous driving in future vehicles? Because obviously you want that to work as fast as possible too.

Don't want to hit the wall.

You want the computer to work fast.

This is something we've been asked a lot. And stay tuned. We want to make sure that everyone has access to AI and there are people who are struggling to get AI working in certain situations and we'd like to be very helpful with them. And having a synchronous and deterministic chip in a driving situation is very helpful. The idea that you put the same data in and you get the same answer every single time, very valuable. And that's actually not true with GPUs. So it makes it very hard in any sort of real time or safety critical situation. And even though that isn't the focus for us in the market, we have built something that's very good for that. And we've had some people come to us and say our focus is that market. Can we do that part? And you provide us the architecture and we go to market on that. I mean, we just want to be helpful to society in that side. It's not our core market, but we would love to help people do that.

Yeah. There's certainly a piece of a robo taxi that is not in the car, like you said. You know, you have to have a system moving cars around and tracking where potholes are on roads and you know, and where crowds are and where, where to move cars to. Stuff like that. Traffic control that could be running on Gro today. Right? You know, it's up in the data center.

Absolutely.

Let's see. Are you annoyed by Elon Musk's name? Elon Musk on X.com, which we're a big fan of, has a AI that's called Grok with a K. You're Grok with a Q. You know what's happening there? You got to sue Elon. Or are you guys going to just keep the both names and deal with it? You know, because I think users are figuring it out.

Well, I, I think what's really important here is that we go by social norms and just gotta say we, we called. Dibs, Dibs, dibs, dibs. We called it first. We called it best. It's ours. We also have the trademark. We sent a cease and desist. We're Grok. And you were Grok first because the.

Other Grock came after you a Long, long time.

And the other guy makes more noise about it, though. Maybe, but that's not how it works. So we've. We've sent Cease and Desist. We. We published a version online, and we are Grok. And there should be no other covered. Dibs. Yep.

All right, so you're going to court, and you'll. The courts will figure it out.

And our hope is that Elon will observe. Dibs. I mean, he's. You know, if he wants to be a standup guy, he can. I think he should understand that. If he doesn't, then we'll have to pursue options. But. Dibs.

I hope you both get to keep your names. I. I think it's fun. I think it helps both of you, but that's me. Yeah, we already covered the technology. You know, why does Llama run better on it that. What. What's.

What.

What are you noticing your customers are doing with Grok?

Well.

Are they running? Do you have a list of favorite models?

It's actually funny. Any model that we release on our hardware, the hardware gets completely saturated. Like, people want to use whatever. Like there's. I won't say which one, but one model that we're running which doesn't get any usage anywhere else, but on us, it gets usage and the speed actually makes that model interesting. But there are favorites. So, for example, when Mixtral came out, it eclipsed Llama 2. When Llama 3 came out, it eclipsed Mixtral. And so people's general preference changes pretty quickly. But the. The thing that. One of the most interesting things was someone. Someone on the social network formerly known as Twitter, I think X now. Sorry, I can't keep track. They posted that when they switched from, I think it was GPT4 to llama 370 billion on Grok, the session duration of their users went from like 18 minutes to 31 minutes, which is a massive increase when you consider that the quality of llama 3, very close to current GPT4, but not quite there. A little short. So clearly that's coming just from the speed.

It's dramatic. When you get a demo of it, it's like GPT is writing like this and you just. Boom. It's like, whoa.

Sometimes I'll be doing the demo and I'll notice that it's done and the person had looked down and, like, you blinked. You missed it. Yeah, it's actually a problem, and then I have to do it again. So, yeah, it is dramatic, but also in terms of applications people are building, things that do require speed. A Lot of the voice stuff is really helped by speed. And you can build something when your latency is a little bit longer, but the engagement is much lower and you see people use it a lot more, the lower the latency. So there's about, I think, five really good services and about 20 in total that use voice that have been built with us. But if you go to our Discord channel, there's an entire channel full of, like, whenever someone builds on Grok, they can post what they built on there. And it's fun to just scroll through all these different apps. It's massive what people have built.

Yeah, it's a lot of fun. Deep Molnar has a weird question. He wants and wants to know what's your favorite programming language and why is it Haskell?

Well, Haskell is the one that I'm generally willing to admit is my favorite program. Not too weird, but let's start off with Haskell and then we'll get into the. The heavy language stuff. So Haskell is one of my favorite languages because I don't like to spend time debugging stuff. Yeah, I like for the. The language to tell me that I made a mistake when I make the mistake. Very often when I'm writing Haskell because of the type system as I am programming, I'm usually asking it to type check every one or two lines of code. And I never look at the error, because when it's every one or two lines of code and you can insert this thing called Undefined in Haskell and it will type check, I don't need it to tell me what the error was. I just type the code. I know what the error is. As long as I know I just made an error so I can program really, really quickly. And the other one is, you know, I. I built some pretty big distributed systems and debugging something like that, very painful. But I remember I would often write C or heaven forbid, Python. And if the unit test passed after writing a bunch of C that compiled, I'd be like, okay, what's wrong here? Because the unit test never passed the first time. In Haskell, it's the opposite. If the unit tests don't pass when it compiles, you're like, wait, what's wrong with the unit tests? Because it compiled, it's probably right. My favorite language is agda. Agda, yes. Okay. Yeah. So as socially ostracizing as Haskell is in being a developer, because it's just such a weird language, that's what AGDA is to Haskellars. It's like the next level up it's is to Haskells where every single function you have Russell, Russell's Paradox. And then things where I'm forgetting which if you insert something and then you look it up later, it's still there unless it's been deleted. Like, and it will not compile. I saw someone do a version where quicksort would not compile unless it was implemented in an. In a law n log n algorithm. Otherwise it would not compile. Not you're checking, it just won't compile unless that's true. So if you really like your code to be correct, AGDA is a fun language to start getting into.

Very cool last kind of questions. Tell me a little bit about your company. How many people are working there now? You?

I think we're at about 220ish people. It actually changes quite often. We had a couple of people join this week, but yeah, we're keeping it lean. And when you think about it, we're doing chips, we're doing systems, we're doing networking, we're doing runtime, we're doing compiler, we're doing orchestration and then we're doing a cloud on top of that.

Yeah.

And we're doing that with 220 people. And we're doing it faster and in a lot of ways better, more energy efficient, so on. And that's the reason why we can do it. If we had more people, we wouldn't be able to do counterintuitive things. Counterintuitive things can only be done by small groups. The bigger your, your, your team, the more what you build is going to look like what everyone else builds. You must keep it small.

I hate committees. I worked at Microsoft, I saw that what committees do to the world, they slow everything down.

Yes. If anyone else is trying to copy us, please create a bunch of committees and grow very large.

Yeah. I met the fourth employee at SpaceX and I was, I said, tell me about the team that lands the rockets on the, on the barges, you know, back on Earth. He's like, that wasn't a team, it was one guy. I'm like, what? How did one guy do all that? He's like, he was working with people, but one guy can keep it all in his head. Right. All, all the controls that need to happen for a rocket to come back to Earth. Right.

It's also more fun because you get to work with a small group of really, really talented people. Like, you know, we have people who scaled AWS from two to 200 data centers. We have the, you know, first CFO of AWS. We have a bunch of early, you know, Google people helping us out. Like these are like the people who have really done things and were there and have the stories, but very humble. That's the key. Because if you hire people with egos, they push out other people who are good. So you have to have a team of fully humble people who are exceptionally good.

I can't work for you because I got a big ass ego. Right. How do you interview for that?

I don't know. The easiest one to interview for. So we really try and hire people who's humble.

I mean, you know, what's the. What are you asking? What do you.

Well, when they ask for a really big title, that's the first clue that they're not humble.

Oh.

And then if they ask for a big team of people, you're like empire builder, probably not humble. You know, are they willing to roll up their sleeves? And then there's some questions I asked that I'm not going to share here that are really telling, but the ease. So we also look for what we call return on luck, which is so many people will have an opportunity land in their lap and they won't, they won't react to it. And so the jokey way that we say we interview for that is we always put candidates in two piles and we throw one away and the one that remains is the lucky pile. We don't really do that.

I got asked all sorts of weird questions when I interviewed at Microsoft and other places can imagine. Humbleness is an interesting attribute that keeps coming up.

It's key.

Yeah, it's hard to be humble sometimes, particularly if you've done things right because that's. That gives you the right to say things right and then.

Yeah, but the way that we say it is the, the way that you've done things in the past before is not relevant. Why you did that in the past is very relevant.

Interesting. Tell me a little bit about the fundamentals. How did you get funded? How did you get the capital to do what you're doing? Because it costs some money to do what you're doing.

Yeah, Chips just cost a couple hundred million dollars. No big deal.

No big. Remember when that used to be an outrageous amount of money in the Valley?

Now it's like worse than that because it's a couple hundred million dollars to get started. And this is where a lot of uh. So there's been over 100 chip startups that have been funded recently. And what most VCs don't understand is it's about $100 million to get started, then you try and ramp into production and then it's billions. Right. And so what most people don't understand, they're getting into this stuff, seeing all these other chip startups and they're like, oh, $10 million great. They're never going to see that money again. Yeah, because they're going to do. That's not even enough to buy a mask. Right. You buy a mask, it's tens of millions for anything new. But the, the thing is there one, there's only going to be a handful that can actually get to the point where they go to product. Right. And your, your goal when you're building one of these companies is to actually deploy it, not just to build a science fair project.

Yeah.

And so it's, it's, it's like the baby turtle thing. There's only a handful of these that are actually going to emerge. Now when we started, it's funny, I actually had zero intention of getting back into semiconductors. I was actually in the Rapid Eval team at Google X. That's the team that comes up with all the crazy ideas of what to do next. So I'd done the tpu, Google, I started that and then I went to Google X and then I started doing things in mechanical engineering, bio, some AI stuff, some more chip stuff, but all over. And I was actually intending to do something a little more around formal reasoning and AI and I'd actually set a little bit of a record for classification model in terms of quality. But one VC asked me, what would you do differently if you had the TPU to do all over again? And I said I'd make the software actually work. And they're like, tell me more. And I'm like, well, it's a nightmare. And I just went through it and they're like, that sounds worth solving. Can you do it? And I'm like, yeah, I'd probably just start with the compiler and they're like, here's $10 million, go do it. So we went through, we got a compiler that worked. We then showed that we could sort of build a chip architecture around that and then we got funded for the rest and then it was off to the races. And that was a fundamental understanding difference. Most people start and design the chip and then they design the software and they say that they're doing software hardware co optimization because they go, oh, we can make some changes in the software based on what we did in the chip, but it's like having a car mechanic design a car instead of a race car driver design the car. You want the race car driver to design the car. I was a software guy. I was all about like, how do I make my life easier? We made a lot of decisions in the chip design that were much harder for the chip designers because it made it easier for the software and compiler. Right. And that's how we got started. And it worked. We got some good results. We've been able to fundraise because of that. And now we have the world's fastest AI chip running off of what is technically ancient silicon fabbed in the US which no one else can do.

It's awesome. These fabs are crazy. I've. I toured Intel's fab up in Oregon and it's, it's definitely the most complex machine I've ever seen. Right. In terms of the scale of the machine, of the building, the amount of machines that are in there building the chips.

You know, each intel lab, they actually put the air ducts in the same place even when they're unused, because they want the environment to be so similar. Because even if there's a physical item in the room, they're afraid that it will affect the way the fab works.

Yeah.

Back in the day, there were, there's a story from Cray back in the day where they discovered that they were having really bad yields of chips. This was early, nowhere near as complicated as today. This was like maybe the 80s or something.

Yeah.

And they discovered that their yields were bad on the days where they made popcorn in the kitchen.

Wow.

Because the, the particles got into the air and actually affected things.

Yeah. My dad used to build semiconductors and took us into the clean rooms. They have to be extraordinarily clean. Right. Otherwise you, you affect yields. Right.

I mean, the wires are smaller than dust particles, so it's like throwing a boulder on there if a single dust particle hits it.

Yeah.

The modern stuff is crazy. Like they, they're not even really using light. EUV is insane. They, they drop these tin pellets, they vaporize them, creates a whole bunch of electromagnetic noise, some of which is more like X ray than euv, but they call it euv. And then it gets re targeted and it hits, it actually emits so much energy that it has, there's risk of it ablating parts of the chip. Like it's almost like a high energy laser, but not light. It's sort of EUV X ray. Ish. And it can actually damage the chip. So there's Crazy physics involved in this.

It's crazy times. Anything that I should have asked you about, about what's going on in the world of AI?

Well, I mean, I think the, the biggest thing that matters to us is our ability to scale.

Yeah.

So going back to that like scaling up thing when, when you look at what it takes to build these chips, there's also all these components that you have to bring in. And if you're building a gpu, you're probably using something called HBM or high Bandwidth Memory. And then you're probably using what's called an interposer or CO os. And then you're probably using super capacitors because the voltage is like a modern GPU from Nvidia uses about twice the power of a house. Right. That's insane. And so the currents in that small little space, like, you know, you can cause all like GPUs fail like crazy because the sort of, the, the high currents and the voltage swings and all this other stuff and the heat. And then if you, if you get past all that, you still have to buy these really high end, scarce interconnects and we've actually built something that doesn't require any of that crazy technology. So all of the HBM for the next year and a half has been bought out. Every GPU that's going to be built has already. The components have already been allocated. There aren't going to be more. Doesn't matter who's building it. Like, you know, if you're amd, your GPU could be better, but if you didn't buy up enough hbm, you're not going to be able to build enough. And so what we've done at Groq, since we don't use any of that, we've built an alternative supply chain for the world, for AI. So we can literally double the world's AI inference compute capacity by the end of 2025 where it otherwise could have been if we didn't exist. And so we're going to get double the AI. But it's not just double the AI, it's going to run faster. You're going to be able to build applications you couldn't otherwise and we're going to do it at lower cost.

Awesome to meet you and talk with you a bit. You're changing the world. So congrats. Where do I.

Thanks.

Yeah, where do I learn more or where do people learn more about Grok? It's Grok with a Q, not with.

A K, Groq with a Q. Go to our Discord channel. We have people there all the time. I drop in occasionally. I haven't told people what my name is on there, but I occasionally answer questions. And if you go to our website Grok with a q dot com, we have the little Discord icon on there. Just click it and feel free to start chatting.