AI Compute & Infrastructure Inference, Training, and the Next Wave of Architectures | AI Rush

Let's try this again. Good afternoon, ladies and gentlemen. Good afternoon. I tell you, I just came from outside. It's a glorious day in London, and I know that we're holding off you guys inside here rather than out there. I'll make it quick and painless, but I want to make a disclosure before we start. I have a disclosure, and you know, do you guys know the difference between an optimist and a pessimist? I am an utmost optimist. And just to basically elaborate, just a very quick note. You know, one day God invites three most powerful people from Earth to a meeting with him: Donald Trump, President of the United States; President Xi, President of China. And Bill Gates, founder of Microsoft. And he says, you guys are the most powerful people on the planet. I invited you. So the bad news is next Thursday I'm going to bring an end to, um, to, to life. Everything will end next Thursday. Go back and tell your people. Donald Trump comes back and says My fellow Americans, a good news and a bad news. The good news is God only invited three most powerful people to a meeting. I was one of them. The bad news is the fact that Thursday is going to bring an end to to war to the world. President Xi comes back, says, "My fellow Chinese people, I have a bad news and a bad news." All these years we believed there was no God. Guess what? There is one. I got invited to a meeting with him, and I met him yesterday. And on Thursday, he's going to bring an end to life. Finally, Bill Gates comes back and says, my fellow Microsoft citizens, I have very good news and another very good news. The first good news: God invited the 3 most powerful people on the planet. I was one of them. Another good news, on Thursday, both Apple and Google will cease to exist. So that's the, you know, when you're really biased and optimist. So what are we talking about? I represent a company, an infrastructure company called Groq. Groq is strictly focused on AI inference. It's a chip and a system that manages inference. And it has gone really viral since 2024. Why? Because it does large language models really, really fast and really, really cheap. The thing that, again, the emergence of DeepSeek highlighted for all of us was the fact that inference is going to be a massive market, a large and growing market, and far larger than training. Again, you can read the data. I'm not gonna be boring and tell you what the slides say, but inference over time is gonna outpace and outdo training by a distant mile. Training is the foundation. And all this that we've heard on these large language models that are very popular and we now use on a daily life is when, again, a lot of CapEx, where a lot of money is spent to train these models. Then inference is when you use these models to produce and to generate money. So if you spend money on training, You make money on inference. And again, these are two, you know, and obviously in our space, infrastructure space, NVIDIA is the undisputed king and queen of training in the infrastructure level, by far the most powerful company on the planet, also a neighbor of ours in Silicon Valley. Now, in training versus inference, runtimes are weeks. The timeframes are much longer. In inference, it's milliseconds. Really, seconds matter. Ultimately, all of agentic things that you've heard here about or you read about or you practice every day, if there is no speed, if there was no broadband, could you and I use Uber? No. Uber would be unusable. It was the advent of broadband which enabled us to use applications like Deliveroo and DoorDash and Uber. Parallelism is enough for training. Low-latency inference is sequential. You cannot parallelize in inference. GPUs are the best for training, are perfect. We tell every customer, For training, go buy NVIDIA. Again, ours is called LPU, Language Processing Unit. And we think— and this is what we're arguing, and luckily a portion of the market believes in what we're doing— LPUs are a better architecture for inference. And again, there is a comment about frequency. This— is my— you know, look at this. I have gray hair. I've been part of the build-out of the internet. I was an executive at Cisco when, you know, Cisco was the hottest company on the planet. And I saw how much excitement and money was expended in building the infrastructure for the internet. This is by far larger and the largest thing that I have lived through and experienced as an individual. Massive dollars. But look how much is spent on data centers and power. Again, Jensen Huang, the CEO and founder of NVIDIA, is heard over and over and over again in conferences and in interviews that it is a power Limited. It's basically the growth of this sector is now limited by power and the optionality of power. And really, we in the United States, and I think United Kingdom, I may be wrong, we're really hitting a concrete wall in terms of access and availability of cheap power. Okay, so alternatives have to be emerging. And again, now you see a lot of investments here and there in France on nuclear. This again talks about the fact that things are large. Look at this. Jonathan Ross is the founder of Groq and the visionary, and he did the original Google TPU, tensor processing unit, and he left and raised a bunch of money to You know, just to be able to snatch or snag a slice of this massive spend and massive market can turn our company or maybe others into very, very large and significant companies on the planet. Of course, all probably, you know, you heard from the previous panel and you've been hearing all day All apps will be AI. You know, even Satya Nadella, CEO of Microsoft, is on record saying end of SaaS as we know it. It's all now agents as a service, not software as a service. That transition is inevitable. And this just came out actually 2 days ago by Sequoia Capital, which is a very prominent venture capital firm. in the Valley, responsible for many of the companies we all use and love and admire. And again, you'd look at the opportunities, emerging patterns. It's all the key areas of AI that all of you or many of you are focused on and are experts in. I'm not, again, going to bore you with. But absolutely no doubt in anybody's mind. We deal with obviously the largest enterprises on the planet from JPMorgan to Goldman Sachs and others, there is so many pockets of development in and around agentic AI and how to use agentic AI to drive business outcomes, to drive business outcomes. The productivities we see now in the programming, coding and programming segment, I mean, it's probably within a year, bulk of the programming that is done in the world is going to be done by AI. We all know that, and this is why we are spending our time here on a sunny Friday. You know, this— okay, some of the slides that were supposed to be hidden started showing up, so no problem. But I just want you to— I'm sorry. Look, it's sort of architectural difference between a GPU-based system and LPU-based system. And again, I am not sure why the animation doesn't work, but we'll figure it out. The GPUs use this thing called HBM, high bandwidth memory. So all these tokens have to basically leave the chip, go into HBM, gets processed, get back in, and then transferred. All of this causes latency and causes what? Power consumption. Power consumption. Look at some of the NVIDIA's announcements in the last GTC last month in San Jose, California. The power consumption of these units are becoming unbearably high and large. Whereas in the LPU-based system, Our language processing unit, our architecture. We execute things in a synchronous and deterministic way and integrated. We don't use any HBM. High bandwidth memory has 2 issues. One, I told you, that causes some latency. But most importantly, the entire supply of the world is done by 2 companies, Samsung and SK Hynix in Korea. And the entire allocation of high-bandwidth memory is sold out for the next 2 years. To whom? NVIDIA. So if your system, your architecture, your approach uses HBM, good luck with your supply chain. We use all SRAM on the chip. The data never leaves the chip. The more memory you need, the more chips you deploy. And all of this is done, again, in a synchronous and deterministic way, which causes high throughput and low cost. Again, power consumption goes down because of not having to deal with high bandwidth memory. Look at the use cases for a GPU/batch type interfaces versus an LPU for real-time inferences. Again, many of you, again, sitting in the City of London where financial services industry is prevalent, you're dealing with fraud systems, early detection of fraud, obviously chatbots in customer service and so on, where milliseconds, milliseconds matter to the user experience and the usability of the system. GPUs are a hindrance. will slow things down. Because the minute you're in the world of GPUs, you're dealing with the world of seconds and in batch, not milliseconds. We're dealing with milliseconds. Again, do it for yourself when I leave or later today. Just go groq.com and try it for yourself. And this is why it has gone viral, because it just runs all of the models that you like, all the large open source and popular models run on Groq and they run really fast, which causes 4 times the speed. Again, these are in average cases. In some cases could be as much as 10 times better, 3 times the efficiency and 5 times the savings. Why the savings? Power consumption. Power. So the key limiting factor for the growth of the AI industry right now. And something, again, I just came back from the Middle East. I had the pleasure of being part of this sort of delegate with Donald Trump visit in the region where also we were part of some of the big announcements from that region. You know, over there we're dealing with 3 cents a kilowatt power cost. And I hate to say when you come to the UK, you're dealing with 30 cents per kilowatt. You cannot be competitive with that level of difference. Again, in Europe, Norway is showing a lot of signs of— and obviously France has a very advanced state of nuclear capability, but UK really needs— and we're meeting with UK officials to address this issue where— What about the UK? Power is a limiting factor for the United Kingdom, which is full of very intelligent people, ambitious people, and great entrepreneurs and great companies that are coming out of the UK left and right, such as Revolut, which we had on the stage a minute ago. This speed with zero marketing budget has led from zero developers on our platform, on Groq Cloud, to more than 1.5 registered developers working and doing their great applications on top of Groq Cloud today. Again, we just recently raised this and publicly announced $640 million round led by BlackRock. But before that, again, you know, low-budget Silicon Valley startup with no marketing budget, with no outbound, you know, demand gen effort, 1.5 million. And you should see what they say in social media and on the internet about their experience with Groq. We're very fortunate that thanks to this development, you know, basically we have hit— as a company, we have hit an inflection point. And my final slide, ladies and gentlemen, in this world you will hear Obviously, people who compete in this space talk, we're the fastest, we're the cheapest. I tell you, I'm in conversations from Japan to Singapore to Brazil to Saudi Arabia and to the United Kingdom. It is not— you have to optimize on 2 dimensions. You have to be fast. And you have to be cheap. These token factories, it's all about token production at the most economical unit and as fast as possible. So you've got to be fast to deliver the user experience and to get answers to the questions that people are asking, but you have to do it cheaply because every day billions and billions and billions of tokens are generated and consumed. Cost is a prohibitive factor. And where the company shines is this optimum quadrant. This is not a quadrant that we drew. I'm getting the signal for the end of time. But again, optimize on both dimensions. And luckily, I am at the end of my presentation and this gentleman is very happy to see that. Again, I don't know whether it was the format of any questions or comments, but I'll be around for a few, and I would be happy to answer any questions. Thank you for the opportunity, and thank you to AI Rose. Thank you.