🚀 AI Insights with Harvard Strategy Expert : History's Third Tech Wave

You're by far one of the most accomplished guests we've had so far on this show. You've been on the board of intel for the longest time. You've been at intel for 28 years.

You've been 29 years. 29 years.

You've been at HBS since 1981 as a member of the faculty. You've written several high selling cases. You've written multiple best selling books, including Strategy Rules, which cover strategy principles from Andrew Groves, Bill Gates and Steve Jobs. You've written the Business of Platforms, two of these books, which I had the good fortune of reading. I had the good fortune of taking the strategy and technology course with you and doing a project following that. You've written among your cases. You've written cases on Apple, you've written cases on Meta, Microsoft, Google, most recently Nvidia. The point is, anything and everything that's happened over the last 30, 40 years in the technology space, you've seen it. And here we are with AI, which is one of the biggest inflection points in this landscape. So what is your advice to business leaders as they think about leading their companies in this environment?

I think it's useful to think about generative AI as being similar in magnitude to the PC in the early 80s and to the Internet in the mid-90s. And I think we can learn a little bit about what do we actually discover about these major discontinuities. And I think there's a couple of very big important lessons. Number one, the discontinuities themselves tend to be of extraordinary importance. However, they tend to be much longer term in terms of rolling out the capabilities that ultimately are embedded in that, in that new technology than most people think. I wrote a book called Competing on Internet time in 1997, 98 on Netscape, and we spent a lot of time with Marc Andreessen where he talked about all the things that the Internet could do and all the things they were building at Netscape and in the 95, 96, 97 period. And Mark was absolutely right. He just happened to be wrong by 20 years. So a lot of the great ideas that he had absolutely materialized, but they didn't materialize for decades. And the same thing is true in the PC space, and the same thing will be true in the gen AI space, which is we know the capabilities that it can ultimately deliver, but actually delivering on those capabilities with a possibility of creating significant revenues and profits is probably going to take a lot longer than most companies who are in the space are willing to admit.

It feels as if the adoption curve of trying this out, because Internet is providing distribution to almost every person is dramatically faster to at least experiment or try. So do you see the gen curve being faster than PC and Internet in terms of adoption by consumers and businesses?

No question. I mean everything is being adopted faster, not just Genai. If you look at, well obviously the emergence of ChatGPT to 100 million users, you saw threads getting to 100 million users even faster. There's no question that the Internet enables much faster distribution. However, having said that, a lot of the issues with the ultimate performance of the PC or even the Internet was not about distribution. It was about building the underlying capabilities that could be reliably delivered at a level of quality and comfort that people were willing to deploy on a large scale. So again, I think it'll be faster than than the earlier adoption curves, but I still think it's actually going to be a little longer before a lot of those capabilities actually develop. So you know, we're seeing today very large numbers of companies are spending enormous amounts of money to experiment with Gen AI capabilities, but the actual deployment tends to be still relatively muted because there's so many questions about whether you can actually deliver those capabilities in a way that don't disrupt the business. And so you see lots and lots of experimentation, which is great and that's going to speed the rate at which adoption ultimately happens. But we are two years into this and the actual deployments tend to be still somewhat muted.

And if you structure the market you can see that there are the infrastructure players, the giants like Nvidia, Microsoft, Oracle, Google, and you have the foundation model companies, the OpenAI, Anthropic, Mixtral and the others. And then you have the startups at the application layer and also quite a lot of these other companies that are trying to implement at the application layer. Where do you see, if you think about two things, one is juror strategy, which you've written a book on and if you think about, look forward and reason back from the perspective of the bigger companies, who do you think is going to have the upper hand and is positioned to win in this scenario?

One of the biggest challenges with Gen AI compared to other technologies, particularly software related technologies, is the capital costs are so much more significant and that puts the larger companies at a big advantage. Just the ability to be able to do training and to then have cloud based services which allow for both the training and the inference certainly gives the bigger companies a bigger advantage than historically they would have had in a software related business. So what's different about Genai is it's really a combination of hardware and software, it's not just a software business. And so the problem for the startups is how do they compensate for the advantages that Microsoft is going to have or even OpenAI has because of all the capital they've been able to raise, as well as Google and all of the big companies who are building out models in this space. Now, I think it's certainly possible that some of the smaller companies, like the anthropics of the world, are going to be able to rely on companies like Amazon, let's say, to actually help them build their capabilities and to build out an alternative to OpenAI, which is very important. But the challenge right now, again, if I'm looking forward and saying what am I going to need in order to get to a level of capabilities that will make me competitive with Google, OpenAI and a couple of the other big players, let's say Llama and Meta, you need to raise an enormous amount of capital. OpenAI has been spectacular at that, and that's going to be the challenge for all the other startups in the space. So I think maybe a couple of them will be able to do it. Anthropic has been pretty successful going down that route. Mistral has struggled, a lot of the others are struggling to raise enough capital to compete. So I think that you end up with a world in which a relatively small number of foundation models, not one, it's going to be a small number, but it's going to be based on people who have the capital to deploy. So then the question is what about the application layer that is more like a software business and therefore still a little bit more capital intensive because of the training requirements. And the inference requirements are still a lot more expensive than the traditional SaaS businesses, but nonetheless they're probably very much within the realm of what a, a startup should be able to do. And I think that there's a lot of opportunities for startups to do things that are unique and distinctive and different and maybe even a little bit more niche like that will keep them safe from the Googles of this world, the Microsofts of the world, and even the OpenAI's of the world.

We're seeing an incredible ability to use lots of inference compute to do completely new sets of workloads that look a lot more like what we used labor for previously. So the amount of total value creation, even if costs are a little bit up, feels like it's on a different order of magnitude. So how should companies think about Adopting tools from folks that are otherwise solving for what a part of their workforce is doing right now.

Well, the critical question is the ability to do automation in a way that delivers the same level of quality without the oversight required from a human labor. And, you know, we certainly have seen in the, the programming area. That's probably where I would guess that the biggest bang for the buck has come in the last 18 months or so. And I expect that is going to just get better and better. A CEO of one of the largest gaming companies was confidently saying to me, I assume he was hopefully right, that he would be able to, within a year or a year and a half, eliminate 30 to 40% of his programmers. And these are very big numbers. Now, the good news is that there's so many jobs for programmers in other spaces. I don't actually think this is a big unemployment problem for the country. I think people actually can be redeployed. But it's going to require 20 changes in skills that are necessary. So on the one hand, the companies have got to figure out where are the areas where they get the highest returns from this kind of automation. There's a couple of areas today that we know they're working very heavily on, mainly on chatbots and mainly on services and capabilities in addition to programming. But a lot of those chatbots and servicing capabilities, in my view, still require a lot of oversight and a lot of human input and a lot of reinforced human learning being added to those models on a continuous basis. So again, it's not so much necessarily we're going to just reduce the total labor. We're going to have to redeploy people and have them do different things.

And on the topic of these models operating at a level that competes with or even replaces human engineers, I had the privilege of reading your upcoming Nvidia case. And in that you talk about the potential of small language models, SLMs, and how they're in a very good position to expand over the space. You also talk about inference being a lot more important than it previously has been. So we're at this point where we have these large models at the moment dominating the space. Everyone, the foundation model companies like to talk about the size of the model. And then there's the benchmarks and there's ELO scores and there's all that. How do you see this playing out with the smaller language models as you know, as the space evolves? Could you share a bit more about that?

Sure. My view is that the large language models will continue to dominate the conversation for another 12 to 24 months, partly because they're getting so much better and we're moving from text based to multimodal. And that requires large language models to do a lot of that compute. So I see that continuing to dominate. In the meantime though, people are investing in these small language models and they're moving things to the edge. The first real example of this is obviously Apple Intelligence trying to move a significant part to the edge. But even they are not willing to say we're going to do it all at the edge. They were saying they have to rely on OpenAI for certain kinds of queries that they know they can accomplish. So it's going to take some time for us to get more and more capable small language models that do things well enough that people are willing to say, I prefer to have something that is smaller, maybe a little bit more confidential, a little less likely to be in the cloud before they actually are willing to toggle in that direction. However, when they do, there are two things that become important. One is the training regimen today, which is dominating Nvidia. Dominating demand for Nvidia ultimately doesn't stay as robust as it is today. It just can't. Nvidia is basically sucking the air out of the entire industry. The whole industry has been decimated by the fact that the large hyperscalers only are interested in buying Nvidia and nothing else. We're going to get to a point in which the models have gotten so big and so competent at the high end that just continuously training those models is going to become less and less efficacious. So people are going to be looking for more task specific models, more highly optimized models that are going to be smaller, maybe on the edge, or can be done on a much smaller set of processors, either accelerators or GPUs, but even smaller GPUs. You don't need a B200 to do a smaller model. And so inference becomes far more important down the road. And the economics of inference has got to get better. It just costs too much today to do it. And there are a lot of ways in which you can reduce the cost of inference. Right? You don't need again a B100 or an H100 or a B200 to do inference down the road. And if you can do it at a much lower cost, then the economics get a lot more favorable for these companies. It reduces some of that trade off between the humans and the cost of trying to do everything with an automated LLM.

Right? And while we're on the topic of Nvidia, your case talks about how Nvidia built Cuda that ended up giving it a reasonable lead in getting to where it is today. The world wishes for alternatives to that. We wish that AMD and Intel were more competitive. And interestingly, Nvidia recently traded places with intel on the Dow Industrial Index. Do you see this as a short term thing? Is this a permanent reversal? How do you see this playing out going forward in the chips industry?

Well, the chip industry is going through a crisis of sorts, just as I was talking about a minute ago, because data center revenues for anything other than Nvidia GPUs got killed last year and this year they're slightly better, but only slightly better. And it's starting to re equilibrate. We're starting to see people now increasingly looking for CPUs again and looking for memory and other kinds of chips that are going to be necessary to build out these data centers. So it will eventually get back to something closer to normal than where we've been in the last two years. In terms of cuda, I think that it has an enormous advantage today because the existing developer community is so used to using it. However, there's no doubt that everyone in the world who is a buyer of Nvidia is looking for alternatives. And there is enormous amount of energy and effort being put into open source models to try to replace cuda. I heard the CTO of one of the largest hyperscalers this week talk about the fact that they're getting better and better at finding ways to replace CUDA in their own internal environments because it's just getting easier to do this programming now. Their expertise is getting deeper. And so my expectation is that while CUDA is going to have a long lasting tale to it, we're going to start to see defections from cuda. It's starting to grow slowly, but over time, inevitably there's enormous investment in trying to find alternatives. So I think Nvidia's issue is, number one, they can't count on CUDA to be their lock in forever. That is going to dissipate over time. And number two, everyone in the hyperscaling business and the data center business wants an alternative to Nvidia just in order to have price competition. They want second sources and they are looking for not only second source GPUs but also looking at CPUs and accelerators that might be able to do the inference at a much, much lower cost point. And that is something that's incredibly attractive both to the Data center as well as obviously the data center customers.

We really see the rise of inference being the only thing that can push Nvidia out within the next 24 month time horizon, if anything can. But we really focused on what we're calling System 2 AI, meaning giving AI systems the ability to think or operate at inference for a long time to get towards precision. So we've been able to drive inference costs down pretty dramatically at the software layers, if you can, could talk a bit about what we're doing at inference and how that might actually affect the path forward.

Yes. So to give you more context, we've invented an orchestration layer above the multi agent frameworks out there, specifically Langgraph, where we're able to have up to 3,000, more than 3,000 AI agents collaborate on solving a specific task. And these are spin up dynamically depending on how complex the task at hand is. So we're able to use these to exceed the limits imposed by the context window. So for example, if you use Claude Sonet, the context window is about 200,000 tokens. If you use OpenAI, it's about 128k. And the more of the window that you use, the more the quality deteriorates in the output. And from an output token standpoint, GPT has about 16k tokens. Claude uses about 8k tokens. Right. So put that in practice. If you use a single AI agent, which there are tools out there, like cursor, replit, agent, GitHub, sparks, which is a new offering, you can produce projects with between 802,000 lines of code. You can go maybe slightly higher than that if you use O1, which is more generous in its limits, but as you go larger, the quality suffers. So the reason we have all of these agents is that we always operate in the optimum space for these models in terms of the context, in terms of the output length, and that helps us get a higher quality output. It's still a very hard problem to solve because code generation in itself, even though programming languages are English like, there are a lot more challenges with dependencies, with bugs, with compiler issues and whatnot that we keep discovering as we encounter new customers. And it's a journey to fix. But we've been able to achieve that and we've been able to keep cost comparatively low. So for a project with about 40,000 lines of code, it takes us about 10 hours of thinking time and two to three hours of code generation time and we produce the output that is, we say about 80 to 90% there. After that, if A single human engineer spends about a week fixing any last mile issues. We cannot get your API keys for you, we cannot set them in the environment. We should not be doing that. You should be setting up your own infrastructure and hooking up code to that. So if you get the human engineer, have them spend a week doing that. You've now built a six month project in about six days. And that's our offering to the world. We're able to do this because the model providers are continuously pushing optimizations and offerings out there. We are big users of prompt caching, we're big users of the infrastructure optimizations that we now have access to. And it's just an exciting time to be building out there if you're at the application layer, just like the startups you mentioned earlier. Because you can now use different model providers to your advantage. And our strategy is to ride on the shoulders of giants as soon as they put out the next big model. A 10x increase in the output context window or Even the input is 100x benefit to us just because we're using all of their context window.

Right, right. Well, the context window has obviously been a huge constraint up until the last couple of generations of models and that's obviously being freed up by the kinds of things that you're describing and it increases the functionality of the models dramatically. So it's a great space to be in. Yes.

Yeah. I think what's so interesting is, you know, being able to drive the cost of something like this from many thousands of dollars on a per run basis. We've been able to drive it down, you know, to a couple hundred dollars on a per run basis through software, through optimization, through optimization, through caching. And so there's not just optimization that can be done all the way down at the chip level from the cpu, but all the way at the, at the application level. So we're sort of stacking S curves of efficiency and we're sort of just on the front end, it feels like for inference, of discovering those where we're maybe towards the middle of the S curve on the, on the other side of the house, of the foundation model house. Anyways, guess on, on how far it goes until we start having to make finishing returns. But it sort of feels like inference is having its coming out moment of saying, hey, this is the next S curve to stack innovations on top of which is really great for startups.

Yeah, I mean, again, you wouldn't want to be in the training part of the world right now. It's just too expensive and Unless you're OpenAI and you can raise $11 billion at the drop of a hat, being at the layer where you're focused on inference, and particularly if you can make the economics more compelling, then it's a much better place to be. And it's a place where startups, as we said at the beginning, have an opportunity to certainly play a much more important role and not be as constrained by the capital requirements that the model providers and a lot of the other players in the space have to deal with.

Right. And while we're talking about cost, and since you've mentioned CPUs for inference, we've historically seen that CPUs are a lot more efficient at running tasks like inference. And you talk about this in your case, where it's magical to have access to these large language models to train them, but the cost that you're running to train the model, and then even at inference, it's much more efficient to run 50 houses for 100 years. You do that and you'd still not even come close to the cost of training a single GPT4 class model. Which is why you think CPUs for inference are one of the solutions to this problem. Or do you see also other solutions to optimize this problem going forward?

Certainly CPUs are one of the most important possibilities for changing the economics of inference. And I know that intel is working on this, AMD is working on this. I'm on the board of Ampere Computing and we're working on this. So everyone's going to try to figure out ways to drive the inference market towards a CPU solution. With some embedded accelerators, you can't just take a CPU off the shelf and do this. But there are a lot of things that I know we are doing at ampere and I'm sure the same thing's happening in AMD and Intel to embed a lot of that capability into the next generation CPUs that will make them much better alternatives again. And part of the problem is the power problem. Just the the power required to run a Data center with 10,000, 20,000, 50,000, 100,000 GPUs is off the map and power has become the biggest constraint. So if you have much lower power CPUs that that can actually be air cooled and not water cooled and be put into an environment where you don't have the same requirements of needing a nuclear power plant to run the data center, which is why Microsoft wants to re engage Three Mile island. Then you're changing the economics in a very dramatic way makes sense.

I'm curious, you're in a lot of small closed room discussions. What is getting talked about in these rooms, you know, at the board level that you know is not confidential but is sort of your surprise is not getting more airtime publicly. Like that's an important strategic issue around AI or AI adoption.

You know, the board discussions tend to be looking out beyond this 12 to 24 month time frame because I think most of us generally feel pretty confident that Nvidia is the solution in that short run. And so that's not the question today. The question is not what are we going to do in the next 12 months. The question is what can we do 24 months and out that potentially changes the nature of the game again. So it's not surprising that the nature of the conversation today is all about Nvidia. Right? That's just what the fact that they took over Intel's place in the DAO certainly gives us an indicator of how the world is thinking about it. And I think it's appropriate. But that doesn't mean that two years from now we're in the same situation. I think that two years from now you can imagine a whole new set of both capabilities that get created in the Genai space, both in the multimodal model, multimodal large language models, the small language models, the edge models, all of these things start to create a new environment. And also just hearing some of these large data centers discuss how they think they can change the game with software that frees them from some of the constraints of CUDA also tells you that looking further out, again, not right now, but looking further out, that it's going to be a much more competitive space down the road.

Well, any closing remarks for the entrepreneurs out there building an AI?

Well, I think this is a wonderful time as long as you make sure you think about the capital that's required, your spending rates and the time frame you have to deliver. Again, what I worry about for a lot of the folks in this space is that 70, 80, 90% of the folks who are in this space today are going to be gone in three to five years. This is very much like the Internet of 1995 to 2000 where people were throwing money at anybody and anything that said Internet on it or dot com, now they're throwing money at anything that says Gen AI. And the reality is if you don't think very, very carefully about the burn rate, the Runway, what capital is required to get to the point before you see some of the changes we've just described you're not going to make it and so it's much more about that meticulous planning around how do you judge the relationship of the resources that you need with the opportunity that is to going going to ultimately come but but maybe is going to take longer than you expect. I think too many of these startups are we're assuming that this stuff was just going to happen like that and it hasn't it's going to just it's going to take time before it gets deeply adopted and then deployed broadly and so you've got you've got to actually plan for that rather than just assume assume it away which I think is the biggest mistake most of the startups are making today. That's awesome.

Things are not a strategy right?

Exactly.

Well yeah I can't imagine a better close from a renowned strategy professor which is your resource allocation is your strategy so thank you so much for your time.

Good luck guys.